Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate issues with repeated runs #5

Open
bethac07 opened this issue Jan 4, 2022 · 0 comments
Open

Investigate issues with repeated runs #5

bethac07 opened this issue Jan 4, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@bethac07
Copy link
Collaborator

bethac07 commented Jan 4, 2022

Runs die partway through the DeepProfiler section after 2-3 new jobs picked up. I suspect the issue is that DeepProfiler is not releasing the GPU somehow. If we're batching at a larger level (ie plate), this is probably fine because we can have one machine per batch, but it's far from ideal.

[ ] Investigate more clearly if it's always failing at the exact same place to see if that gives clues
[ ] See if it's something we can fix on DeepProfiler's side, that would be ideal
[ ] Otherwise, see if we can add a subprocess command to somehow release the GPU

@bethac07 bethac07 added the bug Something isn't working label Jan 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant