Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to shuffle data after each epoch #102

Open
ablaom opened this issue Mar 23, 2021 · 2 comments
Open

Add option to shuffle data after each epoch #102

ablaom opened this issue Mar 23, 2021 · 2 comments
Assignees

Comments

@ablaom
Copy link
Collaborator

ablaom commented Mar 23, 2021

Probably shuffling without breaking batches is fine. I suggest adding an rng hyperparameter as well, which is interpreted as MersenneTwister(rng) if rng is an integer (as elsewhere in MLJ), and falls back to Random.GLOBAL_RNG This could ultimately be passed to the chain intitializers, although Flux does not currently make this easy (FluxML/Flux.jl#1335).

I also suggest the following handling for RNG here (and more generally) to help with reproducibility: The fit method creates a deep copy of the RNG, which then gets mutated as various rand(rng, ...) calls are made. The final state is then output to cache so that update can carry on with the mutated RNG in the case of a warm restart. In this way,

(i) multiple warm-restarts behave the same as training all in one go (modulo the chain initialisation problem), even if the original RNG gets used somewhere else in between restarts; and

(ii) By specifying a concrete RNG at model construction time, cold-restarts (with, eg, fit!(mach, force=true)) give the same behaviour every time.

@ayush-1506

@ayush-1506
Copy link
Member

Will take this up. self-assigning.

@ayush-1506 ayush-1506 self-assigned this Mar 24, 2021
@ablaom
Copy link
Collaborator Author

ablaom commented Jun 14, 2021

@AYUSH-150 We may want to deal with #160 first, as I'm needing this now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants