Training Metrics + History

[prithvi-prakash]

Before I ask my question, I'd like to preface it by thanking all the maintainers of this incredible library. Getting some great results on this for my research!

I have the following questions:

1. During training we see the accuracy history on the validation split; is there a way I could include the accuracy on the training split as well?
2. Is there a way to get the "best performing model" through the training history based on a particular metric?
3. Is there a way to save the plots generated in the end as file? Alternatively, how do I access the training history metrics - loss, accuracy, etc.?

Thank you!

[yangtzech]

1. During training we see the accuracy history on the validation split; is there a way I could include the accuracy on the training split as well?

I don't know. But maybe you can edit the code to add those.

2. Is there a way to get the "best performing model" through the training history based on a particular metric?

See SaveModel from tsai.

See savemodelcallback from fastai.

See here for discussions about built-in callback in fastai if you want further customization.

ps: tsai is based on fastai. So sometimes, you may search about fastai.

3. Is there a way to save the plots generated in the end as file? Alternatively, how do I access the training history metrics - loss, accuracy, etc.?

I did not find. You may add an option to learn.recorder.plot_metrics() to save the picture before show it.

Use CSVLogger from fastai to log the metrics into a file.

Or:

Use learn.recorder.metric_names to get metric names

Use learn.recorder.values to get metrics.

Use learn.recorder.epoch to get epoch numbers.

You can use dir(learn.recorder) to get more information about recorder.

[oguiza]

During training we see the accuracy history on the validation split; is there a way I could include the accuracy on the training split as well?

Yes. When you use ts_learner, you just need to pass train_metrics=True. When the model is trained, you will see the same metrics for the training and validation sets.