Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken global state between Hydra runs #574

Open
rsokl opened this issue Nov 8, 2023 · 4 comments
Open

Broken global state between Hydra runs #574

rsokl opened this issue Nov 8, 2023 · 4 comments
Labels
bug Something isn't working hydra/omegaconf Tied to behavior of hydra/omegaconf

Comments

@rsokl
Copy link
Contributor

rsokl commented Nov 8, 2023

I don't know what is going on here. It only occurs for pre-release, and re-running the job often fixes it. It occurs in tests/test_launch/test_validation.py and is caused by:

 hydra.errors.MissingConfigException: In 'hydra/config': Could not find 'hydra/sweeper/basic'

@jgbos any ideas?

@rsokl rsokl added test suite Changes to hydra-zen's test suite help wanted Extra attention is needed labels Nov 8, 2023
@jgbos
Copy link
Contributor

jgbos commented Nov 13, 2023

That is weird. I'll try to look at it this afternoon.

@rsokl
Copy link
Contributor Author

rsokl commented Nov 27, 2023

@jgbos I think there is a real issue here, although I'm unsure what the culprit is. I'd like to get a fix in for launch before v0.12 is released. Any chance you'd be able to take a look this week?

@rsokl
Copy link
Contributor Author

rsokl commented Dec 7, 2023

I am actually less sure that this is a "real" issue in that I do not think it is affecting every day use of launch. This is a very weird issue that might have something to do with hydra and pytest? Not sure. Anyway, I am going to release v0.12 regardless as this issue certainly is no different in v0.11 than it is in v0.12.

@rsokl
Copy link
Contributor Author

rsokl commented Dec 12, 2023

This was really hard to repro -- still not sure the circumstances that cause this vs not -- but there is some broken statefulness in Hydra between runs which causes Hydra's store to misbehave with the basic sweeper. Fortunately there is a simple fix - running the following clears ~all global state from Hydra!

from hydra.core.singleton import Singleton

# this should go in a `finally:` after each launch call
Singleton._instances.clear()

Because this clears all global state, Hydra will populate everything anew on subsequent runs.

We should clear state this way within launch and zen.hydra_main. This would have to wait for v0.13 as it is potentially compat-breaking (but only in the most weird ways?)

@rsokl rsokl added bug Something isn't working hydra/omegaconf Tied to behavior of hydra/omegaconf and removed help wanted Extra attention is needed test suite Changes to hydra-zen's test suite labels Dec 12, 2023
@rsokl rsokl changed the title pre-release CI job is flaky Broken global state between Hydra runs Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working hydra/omegaconf Tied to behavior of hydra/omegaconf
Projects
None yet
Development

No branches or pull requests

2 participants