Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Training stalls at the end of the first epoch with keras 3.6.0 #463

Open
IgorTatarnikov opened this issue Oct 9, 2024 · 2 comments
Open

[BUG] Training stalls at the end of the first epoch with keras 3.6.0 #463

IgorTatarnikov opened this issue Oct 9, 2024 · 2 comments
Assignees
@IgorTatarnikov
Labels
bug Something isn't working

Comments

@IgorTatarnikov
Copy link
Member

Training seems to enter an infinite loop, or wait for a callback that's not signalled at the end of an epoch. This causes training to stall at the end of the first epoch.

See https://github.com/brainglobe/cellfinder/actions/runs/11209740840

Downgrading keras to 3.5.0 fixes this issue. #462 pinned this version temporarily.
@IgorTatarnikov IgorTatarnikov added the bug Something isn't working label Oct 9, 2024
@alessandrofelder
Copy link
Member

@IgorTatarnikov has volunteered to investigate - thank you!

@IgorTatarnikov IgorTatarnikov self-assigned this Oct 17, 2024
@IgorTatarnikov
Copy link
Member Author

keras-team/keras#20425 Should be fixed in keras main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@IgorTatarnikov IgorTatarnikov

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alessandrofelder @IgorTatarnikov