Unscheduable garbage collection job and how I solved it

[SISheogorath]

The other day I had a garbage collection job that was never picked up by a worker from the jobservice. All other queues worked fine, even scheduling one worked, but it was never picked up.

Since I never found the answer despite seeing some issues showing similar problems e.g. #20029, #19494 (comment), #18900

I dug down and found the problem: A stale lock for the job in redis.

Caution

This is low level tinkering with your setup and might break it, follow at your own risk. Please use your brain, thanks! 🧠

So If anyone experiences that a job is stuck or not picked up, like in my case the GARBAGE_COLLECTION, here is how I solved it:

1. Stop all jobs
2. Restart the job service to make sure there are no more jobs running
3. Connect redis in the namespace of your jobservice (see your harbor config for details, it was redis-cli -n 1 for me)
4. Find all garbage collection keys using keys *GARBAGE_COLLECTION*, if it's broken the same way as mine, it looks like this:

   127.0.0.1:6379[1]> keys *GARBAGE_COLLECTION*
   1) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION"
   2) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock_info"
   3) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:19b507bb4308eac2e7711356:inprogress"
   4) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:max_concurrency"
   5) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock"
   

5. Check if the job is locked: GET "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock" it should return "1" if it's locked (which is the origin of the problem)
6. Delete the unnecessary inprogress entry (remember, we stopped/restarted the job service to make sure nothing is running): DEL "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:19b507bb4308eac2e7711356:inprogress"
7. Unlock the job using: SET "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock" "0"
8. Schedule a new garbage collection job and see it run 🏃🏻

Since this topic seems to come up similarly ever now and then, I couldn't find the answer in any of the threads, I hope this is a good place to provide this info.