Way to remove task from queue that is in "allocating" state

[aka-Ani]

Sorry if bringing this up as an issue here is incorrect, but I was wondering if there is a way to remove a job from the queue that has not yet started but is in the "allocating" state. I can't seem to -k it because it says the job is not finished or is not running, and I can't -r it either as it says the job cannot be removed. Thanks!

[justanhduc]

Hi @aka-Ani. I just wanna check whether your job is the first in the queue and for some reason it cannot be executed. In that case yes, -r doesn't work. If it's not the case then could you please let me know how to reproduce the issue?

[aka-Ani]

Hmm now that you mention it, it actually was the first in the queue, and it was not executed yet because the GPUs were in use.
I tried to see if it would happen again with three 1 GPU jobs in a 2 GPU machine, but it worked as expected and I was able to -r to remove a job in the "allocating" state.

The situation when I encountered this issue was when the machine was already using the GPUs from another script (not through ts) so the very first job submitted to the task spooler was in the "allocating" state, presumably waiting for the GPU memory to free up. But I wasn't able to remove this job from the queue or swap priority with another submitted job. I just had to wait till the non-ts script finished, and task-spooler began executing the queued jobs.

I maybe won't run into this again if I make sure to use task-spooler for all scripts that use the GPU resources. Thanks for the response on this, and for making this useful tool!

[justanhduc]

Hi @aka-Ani. Thanks for raising the issue. I'm already aware of this problem, but didn't find it annoying enough to fix 😅. I will look into this and push a fix in the next version so I will keep this issue open. Thanks a lot for using ts!