please explain scheduling policy (job/node quotas?)

at the moment it seems quite impossible to predict availability of nodes for testing.

(at the moment, the only info is that job pairs are "enqueued". Then nothing changes - I guess nodes are busy.)

Can you improve this somehow?

* partition the nodes, and assign to communities?
* on the assigned nodes, jobs by community members take precedence (but others can run, especially if the nodes were otherwise idle)

* introduce some other system of prioritizing the assignment of jobpairs to node that favours running small jobs (few benchmarks, short timeouts)

it would already be a great help if you make the current policy and cluster status more transparent.

Hello, Johannes. The current scheduling policy is that periodically (every 30 seconds or so) all jobs which are to run on a particular queue (let's say all.q for example) will all get a modest number of job pairs added to the queue, if the queue is not too full (around 800 job pairs, I believe). So all jobs on the queue will make progress simultaneously.

Now unfortunately, sometimes what you are seeing on the cluster status page does not reflect what is really happening on the cluster. This is not good, but let me explain why. The cluster status page shows StarExec's view (from the database) of which job pairs are currently enqueued with Grid Engine. When a job pair starts running on a compute node, the first thing it does is write back to the database to update its status from enqueued to running. If something goes wrong and it cannot do that successfully (for any number of reasons), then StarExec will still think that job pair is enqueued.

So it has happened sometimes that it looked like all.q is completely bogged down and jobs aren't making progress, when really the cluster is idle and StarExec's view of what is happening is wrong. We do need a more robust way to detect when this has happened. When we do detect this, though, it pretty much requires manual intervention. So we need something to detect this and send me an email. An alternative is we could just dump the output of the Grid Engine qstat command, which shows what is really happening with Grid Engine, and then eager users would notice that nothing is running but the cluster status page shows lots of job pairs enqueued. Then those users would let me know (if I had not noticed myself). Because I am aware of this issue and it has come up several times, I am usually checking daily to make sure that work is happening on StarExec if the cluster status page says that job pairs are running.

Aaron

Admin wrote:(let's say all.q for example) will all get a modest number of job pairs added to the queue, if the queue is not too full (around 800 job pairs, I believe).  So all jobs on the queue will make progress simultaneously.

It seems that this number (800) is fixed and does not depend on the queue size. How about choosing this dynamically, e.g., 5 times the number of nodes in the queue? This would help for the execution of small jobs (e.g. currently all.q has 7 nodes and 800 jobpairs queued--so new jobs start with a significant delay).

Harald.

Hi, Harald.

Sorry about this. You are right: the queue size is currently fixed at this large number (800), when it should probably be some kind of factor which we multiply the number of nodes in the queue by. We will change this.

Aaron

Hi,

According to [1] the queue size now depends on the number of nodes in the queue. I spotted the following behaviour (for a single node queue).
5 jobpairs are enqueued, they are handled immediately but it takes 5 minutes until the next job pairs are enqueued. So the queue is most of
the time empty and the node idle ...

I now see why the queues have been fairly large before. Sorry for having made the proposal with (too) small queue sizes---I did not know that
jobpairs are enqueued in 5 minute intervals only. Can this happen e.g. every minute? Or can you enqueue (significantly) more than 5 problems
to an almost empty queue?

Edit: Sorry, this problem should be independent of the queue size. If initially too few jobpairs are enqueued then the queue never gets full.

Harald.

[1] http://starexec.wordpress.com/2014/06/30/redeploying-job-explorer-rerunning-job-pairs-queue-sizing/

Job pairs are supposed to be added to the queue every 30 seconds, but looking at the logs yes, I do see that for some reason it is every 5-6 minutes or so. Sorry about that. I will investigate.

It is a bit of a balance between responsiveness to new jobs and throughput, in setting these parameters. We can keep tweaking them (possibly making this community-settable?).

Aaron

Admin wrote:Job pairs are supposed to be added to the queue every 30 seconds, but looking at the logs yes, I do see that for some reason it is every 5-6 minutes or so.  Sorry about that.  I will investigate.

It is a bit of a balance between responsiveness to new jobs and throughput, in setting these parameters.  We can keep tweaking them (possibly making this community-settable?).
Aaron

I never spotted a problem before, so I guess things are fine again once the 30 seconds interval is in place again.

Harald.