You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Runs die partway through the DeepProfiler section after 2-3 new jobs picked up. I suspect the issue is that DeepProfiler is not releasing the GPU somehow. If we're batching at a larger level (ie plate), this is probably fine because we can have one machine per batch, but it's far from ideal.
[ ] Investigate more clearly if it's always failing at the exact same place to see if that gives clues
[ ] See if it's something we can fix on DeepProfiler's side, that would be ideal
[ ] Otherwise, see if we can add a subprocess command to somehow release the GPU
The text was updated successfully, but these errors were encountered:
Runs die partway through the DeepProfiler section after 2-3 new jobs picked up. I suspect the issue is that DeepProfiler is not releasing the GPU somehow. If we're batching at a larger level (ie plate), this is probably fine because we can have one machine per batch, but it's far from ideal.
[ ] Investigate more clearly if it's always failing at the exact same place to see if that gives clues
[ ] See if it's something we can fix on DeepProfiler's side, that would be ideal
[ ] Otherwise, see if we can add a subprocess command to somehow release the GPU
The text was updated successfully, but these errors were encountered: