How to Restore Best Model

[skumarlabs]

Hi Team,

I trained an AR model as below

hyperparams = {
        "n_lags": 2,
        "n_forecasts": 1,
        "epochs": 50
}
set_random_seed(314272)
m = NeuralProphet(**hyperparams, accelerator='cpu')
train_metrics = m.fit(df_train, validation_df=df_val, early_stopping=True, checkpointing=True, freq="D", progress="bar")

and I got below train metrics after early stopping stopped the training after 29th Epoch.

MAE_val	RMSE_val   Loss_val   RegLoss_val	epoch	    MAE	         RMSE	Loss	RegLoss
0.6882	1.6068	    0.3633	    0.0000	    0	    0.5478	0.7176	0.1885	0.0000
0.3918	1.3110	    0.1678	    0.0000	    1	    0.3707	0.4975	0.0932	0.0000
0.2617	1.1657	    0.1140	    0.0000	    2	    0.2211	0.3200	0.0380	0.0000
0.2270	1.1029	    0.1018	    0.0000	    3	    0.1480	0.2390	0.0208	0.0000
0.2134	1.0651	    0.0963	    0.0000	    4	    0.1225	0.2091	0.0158	0.0000
0.2024	1.0382	    0.0920	    0.0000	    5	    0.1112	0.1935	0.0135	0.0000
0.1984	1.0262	    0.0901	    0.0000	    6	    0.1040	0.1835	0.0123	0.0000
0.1988	1.0247	    0.0902	    0.0000	    7	    0.1007	0.1765	0.0114	0.0000
0.1966	1.0203	    0.0892	    0.0000	    8	    0.0982	0.1717	0.0108	0.0000
0.2006	1.0330	    0.0918	    0.0000	    9	    0.0967	0.1681	0.0104	0.0000
0.1989	1.0256	    0.0907	    0.0000	    10	    0.0962	0.1662	0.0102	0.0000
0.1999	1.0297	    0.0916	    0.0000	    11	    0.0957	0.1650	0.0100	0.0000
0.1961	1.0221	    0.0901	    0.0000	    12	    0.0963	0.1643	0.0100	0.0000
0.2099	1.0465	    0.0949	    0.0000	    13	    0.0960	0.1638	0.0099	0.0000
0.1985	1.0303	    0.0911	    0.0000	    14	    0.0967	0.1646	0.0100	0.0000
0.1994	1.0225	    0.0905	    0.0000	    15	    0.0968	0.1630	0.0099	0.0000
0.2130	1.0492	    0.0953	    0.0000	    16	    0.0963	0.1627	0.0099	0.0000
0.2093	1.0389	    0.0938	    0.0000	    17	    0.0969	0.1642	0.0100	0.0000
0.2093	1.0364	    0.0930	    0.0000	    18	    0.0961	0.1633	0.0099	0.0000
0.2193	1.0539	    0.0967	    0.0000	    19	    0.0965	0.1630	0.0099	0.0000
0.2063	1.0405	    0.0930	    0.0000	    20	    0.0968	0.1635	0.0100	0.0000
0.2166	1.0534	    0.0965	    0.0000	    21	    0.0962	0.1623	0.0099	0.0000
0.2090	1.0418	    0.0940	    0.0000	    22	    0.0963	0.1630	0.0099	0.0000
0.2202	1.0443	    0.0960	    0.0000	    23	    0.0957	0.1617	0.0098	0.0000
0.2211	1.0466	    0.0966	    0.0000	    24	    0.0949	0.1614	0.0097	0.0000
0.2275	1.0632	    0.0994	    0.0000	    25	    0.0947	0.1609	0.0096	0.0000
0.2294	1.0722	    0.1008	    0.0000	    26	    0.0943	0.1610	0.0096	0.0000
0.2225	1.0498	    0.0964	    0.0000	    27	    0.0939	0.1597	0.0095	0.0000
0.2138	1.0507	    0.0958	    0.0000	    28	    0.0935	0.1597	0.0095	0.0000

I can see that the best epoch is #8 with Loss_val=0.0892 but when I test the model I get Loss_val=0.0958 i.e. the model after last epoch which is not optimal. Could you please help me restore the model from 8th Epoch? SInce I have enabled the checkpointing, I do see files ['last.ckpt', 'epoch=8-step=2178.ckpt'] but I am not sure how to restore using these checkpoint files.

PS: I am aware that I can retrain the model only till 8th epoch to get the optimal model state. But I wondering since we already have checkpoint, if we can use that?

[leoniewgnr]

Did you try this: https://lightning.ai/docs/pytorch/stable/common/checkpointing_basic.html