Job not running even though queue is displayed as empty

Heya,

I'm not sure if this qualifies as a bug, but here we go anyway: I (mabrocks / id 110) have set up a new job (id 6277) in all.q. From what I can see through the cluster status, the nodes in that queue are idling (i.e., no jobs are shown, and the nodes are indeed reporting a load of 0.0* in the qstat output). However, my job is not run. I've tried pausing / resuming the job, as well as deleting and re-uploading a new job. I've also tried to move the job to all2.q, where it was not executed as well. Any idea what is going on? I could successfully run things yesterday / this morning...

Marc

Hi, Marc. Hmm, sorry you are having this problem. Let me investigate and see what's going on.
Aaron

Hi, Marc. Ok, due to a problem that it is going to take us a little while longer to get to the bottom of, StarExec was stalled out and not running anybody's jobs. We got it unstuck, and your job has completed now.
Sorry for the delay,
Aaron

We tracked down the bug, and will be deploying a fix tomorrow. Part of the problem was that since switching to running two jobs per node (thus doubling StarExec's compute power!), some defensive mechanisms we had in place for when job pair's results could not be reported back in to the database (which happened occasionally due to transient system issues, it seems) -- this mechanism was not enabled. That led to another issue being exposed, in our scheduling algorithm. These problems were not easy to find, but very easy to fix.
Aaron

Hi Aaron,

thanks for tracking this down and taking care of the system!

Marc

The fix has been deployed now, so hopefully this won't happen again.
Aaron