StarExec
Would you like to react to this message? Create an account in a few clicks or log in to continue.

StarExec display, and smt-comp

Go down

StarExec display, and smt-comp Empty StarExec display, and smt-comp

Post by mdeters Thu Jun 12, 2014 12:41 pm

As I understand, SMT-COMP will use a sum of CPU times as the "time" for a solver in a division, excluding timeouts, incorrect results, and other failures. This differs from the StarExec display in at least two ways:

1. StarExec shows the wall-clock time for job pairs. This doesn't work for e.g. a multithreaded solver, where a job pair that times out may appear with 750sec time whereas a single-threaded one appears with 1500sec. The scatter and cactus plots are similarly affected, giving a skewed result (if one is trying to compare results in the way that SMT-COMP is to compare them).

2. StarExec sums up all times, including timeouts and (I presume) incorrect results. So the time summation on StarExec does not match what has been used in previous SMT-COMPs (and reading the rules for 2014 I'm a little unclear on the 'm' time component of the <e,n,m> triples in the main track and whether this represents a change from previous years).

Of course enough information is in the CSV to reconstruct the results, but the display makes it hard to judge things like winners within a logic or benchmark family. Is there to be a setting in the StarExec display to allow (1) which kind of time, wall or CPU, is displayed, and (2) how the results are summed?

mdeters

Posts : 11
Join date : 2014-06-03

Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum