StarExec display, and smt-comp
Page 1 of 1
StarExec display, and smt-comp
As I understand, SMT-COMP will use a sum of CPU times as the "time" for a solver in a division, excluding timeouts, incorrect results, and other failures. This differs from the StarExec display in at least two ways:
1. StarExec shows the wall-clock time for job pairs. This doesn't work for e.g. a multithreaded solver, where a job pair that times out may appear with 750sec time whereas a single-threaded one appears with 1500sec. The scatter and cactus plots are similarly affected, giving a skewed result (if one is trying to compare results in the way that SMT-COMP is to compare them).
2. StarExec sums up all times, including timeouts and (I presume) incorrect results. So the time summation on StarExec does not match what has been used in previous SMT-COMPs (and reading the rules for 2014 I'm a little unclear on the 'm' time component of the <e,n,m> triples in the main track and whether this represents a change from previous years).
Of course enough information is in the CSV to reconstruct the results, but the display makes it hard to judge things like winners within a logic or benchmark family. Is there to be a setting in the StarExec display to allow (1) which kind of time, wall or CPU, is displayed, and (2) how the results are summed?
1. StarExec shows the wall-clock time for job pairs. This doesn't work for e.g. a multithreaded solver, where a job pair that times out may appear with 750sec time whereas a single-threaded one appears with 1500sec. The scatter and cactus plots are similarly affected, giving a skewed result (if one is trying to compare results in the way that SMT-COMP is to compare them).
2. StarExec sums up all times, including timeouts and (I presume) incorrect results. So the time summation on StarExec does not match what has been used in previous SMT-COMPs (and reading the rules for 2014 I'm a little unclear on the 'm' time component of the <e,n,m> triples in the main track and whether this represents a change from previous years).
Of course enough information is in the CSV to reconstruct the results, but the display makes it hard to judge things like winners within a logic or benchmark family. Is there to be a setting in the StarExec display to allow (1) which kind of time, wall or CPU, is displayed, and (2) how the results are summed?
mdeters- Posts : 11
Join date : 2014-06-03
Similar topics
» starexec.org is down
» (minor bug) in new display of job pairs
» Starexec logo (for Floc Olympics certificates)?
» StarExec back up
» Forum for StarExec has moved
» (minor bug) in new display of job pairs
» Starexec logo (for Floc Olympics certificates)?
» StarExec back up
» Forum for StarExec has moved
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum
|
|