Separate stdout and stderr

Currently, the job output conflates messages output on stdout and stderr. It would be better to keep the two separate, and only pass standard output to the post-processor (or perhaps make error messages available in a separate file, passed as $2).

Rationale: Error messages vary in format and contents. Occasionally, they may (and, at least in the case of SMT-EXEC, they did) accidentally contain words that happen to be identical to valid solver responses, particularly when these are short strings ("sat") or common words ("unknown"). At present, post-processors have no reliable way to distinguish between the two.

in principle I agree, although it's a bit late now (to change this for the July competitions).

here's another suggestion: post-processors have to process the output anyways - to strip away the timing info at the start of each  line. So, this extra info could also contain an indication of where that line came from. (that should be easier to implement on starexec - easier than storing and showing two output files instead of one.)

OTOH, I think a postproc that just greps for "sat" is, well, somewhat broken. Or the spec that allows this kind of solver output, is. For termination, we require YES/NO as the exact contents of the first output line (later lines are ignored). And indeed, because of starexec behaviour, we changed this to "first line of merged stdout/stderr". I don't like it, as it complicates tracing - solvers have to cache what they want to write to stderr.

j.waldmann wrote:in principle I agree, although it's a bit late now (to change this for the July competitions).

I am not suggesting to implement this right now. But I assume that StarExec will continue after the summer.

here's another suggestion: post-processors have to process the output anyways - to strip away the timing info at the start of each  line. So, this extra info could also contain an indication of where that line came from. (that should be easier to implement on starexec - easier than storing and showing two output files instead of one.)

That is not such a bad idea. It would perhaps more likely break existing post-processors than adding an argument, but I don't think this is a big deal either way.

OTOH, I think a postproc that just greps for "sat" is, well, somewhat broken. Or the spec that allows this kind of solver output, is. For termination, we require YES/NO as the exact contents of the first output line (later lines are ignored). And indeed, because of starexec behaviour, we changed this to "first line of merged stdout/stderr". I don't like it, as it complicates tracing - solvers have to cache what they want to write to stderr.

For SMT, there is an existing specification (written before StarExec was implemented) of the expected output. Unfortunately, this is more involved than "the first output line is YES/NO." In a perfect world, StarExec would allow us to write a post-processor that detects whether solvers provide correct output according to this existing specification.

Hi.

I agree that the design goal for starexec should be to allow communities to implement their (existing) specs easily.

I think starexec's goal is to use existing machinery for this (that some communities may have been using all along).

I think starexec uses runsolver http://jsat.ewi.tudelft.nl/content/volume7/JSAT7_12_Roussel.pdf http://www.cril.univ-artois.fr/~roussel/runsolver/

The paper does not mention the stdout/stderr issue, but the source code does, see comment in runsolver.cc before #undef TIMESTAMPSEPARATESTDOUTANDERR , copied below

// if TIMESTAMPSEPARATESTDOUTANDERR is defined, when time stamping the
// solver output, stdout and stderr are transmitted separately to
// runsolver. This lets us print whether the line comes from stdout or
// stderr. Unfortunately, when streams are separated, we have no
// guarantee to get the lines in the order they were sent (try with
// testtimestamper.cc). For this reason, this flag should be
// undefined. When undefined, stdout and stderr are combined together
// in the solver and are transmitted through one single pipe to
// runsolver. Anyway, this is the usual way to pipe a program output
// to another program input

So I guess TIMESTAMPSEPARATESTDOUTANDERR should be defined?

(I can see no reason why "the order they [the lines] were sent" would be important for competition.)

For reference, here a a few lines of output from 'runsolver' with the macro defined:

...
e0.11/0.13 #clauses of length 1: 98
e0.11/0.13 #clauses of length 2: 2574
e0.11/0.13 #clauses of length 3: 2436
e0.11/0.13 #clauses of length 4: 776
e0.11/0.13
e0.11/0.13 Starting solver
o0.11/0.13 YES
o0.11/0.13 system minus (x, 0) -> x
o0.11/0.13 minus (s (x),
o0.11/0.13 s (y)) -> minus (x, y)
o0.11/0.13 f (0) -> s (0)
o0.11/0.13 f (s (x)) -> minus (s (x),
...

the post-processor can then filter the lines by inspecting the first char (e/o).