Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperparameter sweep "crashed" logging is false #79

Open
kylebgorman opened this issue Dec 27, 2024 · 1 comment
Open

Hyperparameter sweep "crashed" logging is false #79

kylebgorman opened this issue Dec 27, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@kylebgorman
Copy link
Contributor

If a hyperparameter sweep run uses patience to stop, you'll see something like the following logged:

sweep.py INFO: 27-Dec-24 01:15:00 - Monitored metric val_loss did not improve in the last 10 records. Best score: 0.543. Signaling Trainer to stop.

However, W&B treats this is as a "crash" rather than a successful halt. This is strange. Perhaps when patience halts training, it returns a non-zero error code. Assuming this is a unique error code and different than the error code used for actual crashes, we could intercept it here to eliminate this.

@kylebgorman kylebgorman added the bug Something isn't working label Dec 27, 2024
@kylebgorman kylebgorman self-assigned this Dec 27, 2024
@kylebgorman
Copy link
Contributor Author

My hypothesis about what is happening is wrong. Early stopping doesn't give you a non-zero error code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant