[FEATURE REQUEST] A way to save the scan after every permutation tested

[JohanMollevik]

It would be great to be able to save the scan to a file after every premutation to be able to continue where it left of if it was interupted for some reason. Right now if I have a long running scan process and the computer shuts down I have to redo everything. If we could save and continue scans this would allow backing up the partial results and avoid loosing as much work.

[mikkokotila]

I agree, this is a key feature to add. IMO the single most important entirely new feature to add. It will be straightforward to do this with the building blocks that are already there (I think). I will work on this and it will be included in the v.0.5 which should be a few weeks away.

[renatobellotti]

I would like to make a suggestion for this. I'm not sure if you would consider this a different feature, but it might be at least a quick temporary fix.

• Scan() creates a directory talos_output
• For each experiment, Talos creates a subdirectory, named after the ID of the experiment (the row number in the .csv file)
• Scan() adds the path to the experiment's output directory to the params dict before passing it to the model function.
• The model function can now save everything to this directory, and it is clear which experiment the output belongs to.

Would this be feasible?

[mikkokotila]

I think that does make sense.

Regarding the specific case that @JohanMollevik is suggesting, given we already store the log with information on each permutation, we would have to decide what else is a must have in the folder. Maybe start by storing the weights and json for each model, as together with the results that should be enough to not lose anything substantial even though there was a system crash or something like that. That part would have to be an argument in Scan() though, maybe model_backups=True.

[renatobellotti]

I think it might also be nice to have some plots of the history during training (of all metrics & the loss), and perhaps some explanation plots (SHAP? Lime?)

But most important for reproducible runs is the state of the used RNGs. That should be logged definitely.

[mikkokotila]

I think it might also be nice to have some plots of the history during training (of all metrics & the loss)

This is now available with ExperimentLogCallback

[mikkokotila]

More updates regarding this:

• Scan(...experiment_name...) is now required for all experiments
• in the present working directly, folder with that name will be created
• inside that folder the experiment log is stored with .csv ending using second precision timestamp
• if using ExperimentLogCallback one must use the same experiment_name there as input
• ExperimentLogCallback will do matching for the actual file name based on the most recently modified file in that folder, which should be the experiment log
• experiment log with permutation data has .csv ending and epoch-by-epoch log has .log ending

EDIT: One of the most obvious advantages here is that experiments remain cleanly in their own folders, and there is no need to remember to change the name of the experiment between iterations.

This should be pushed later today.

[mikkokotila]

Update

From 0.6.3 onwards, there is available Scan(...reduction_method='gamify') which outputs to the experiment folder with the experiment id a .json file during each permutation. In that file there is a dictionary with dictionaries, where the main dictionary keys consist of hyperparameter labels, and the sub-dictionaries consist of hyperparameter values corresponding to the label (e.g. lr and 0.01). Together with the value there is a string 'active' by default. Changing this to 'inactive' in the local file, during the next permutation all permutations in the parameter space with that value will be dropped.

Related with this, as of yesterday a visual command center (well for now it's just the visuals, but commands are coming). It's called Gamify and it allows you to visualize and analyze the experiment results during the experiment through a web browser.

With the Talos v.0.6.3 changes, also controls will be added to Gamify, so that you can analyze the results, make changes, and see the changes for yourself in real-time. Gamify has live monitoring for epoch-by-epoch data as well.

[mikkokotila]

I'm closing this as this now lives in Gamify.

[renatobellotti]

Thank you very much!