Summary of Future Development

[mikkokotila]

Energy Draw

• Make table which pulls the data from here
• Add energy draw total for each permutation
• Add the total to the experiment_log table

Experiment results API

• Create a Data-API which returns data from the centralized data store
  • By metric (e.g. val_acc) for all permutations
  • By metric (e.g. val_acc) for all permutations filtered by hyperparameter-value pair (e.g. val_acc for permutations where optimizer==adam)
  • By loss (e.g. val_loss) for all permutations
  • By metric (e.g. val_loss) for all permutations filtered by hyperparameter-value pair (e.g. val_loss for permutations where optimizer==adam)

If it will be rest API, I think we should use FastAPI. If instead we want to create GraphQL API, then I think we should use Hasura.

NOTE: The basis for this API is that it always shows the all the hyperparameters for each permutation together with the given result.

Experiment Dashboard

• Create barebones GUI
  • "Card" which show number of nodes running
  • "Card" which shows number of permuations completed
  • "Card" which shows time spent per permutation
  • "Card" which shows total time spent
  • Line plot with two dropdown selectors:
    • First selector is for metric (shows all that are used in experiment)
    • Second selector is for loss (shows all that are used in experiment)
  • Another line plot with four drop down selectors:
    • First selector is for metric (shows all that are used in experiment)
    • Second selector is for loss (shows all that are used in experiment)
    • Third selector is for hyperparameter name (shows all that are used in experiment)
    • Fourth selector is for hyperparameter value (shows all that are used in experiment for the selected hyperparameter name in the third selector)

Experiment Dashboard (second phase)

• Make it possible to click through to a permutation and see the training history for that particular permutation
  • One "Card" per hyperparameter-value pair
  • Line plot with two dropdown selectors:
    • First selector is for metric (shows all that are used in experiment)
    • Second selector is for loss (shows all that are used in experiment)

NOTE: here the plot will show epoch-by-epoch progression for a single permutation.