which value is recommended to be reported as Intermediate Result?

[JianJuly]

Is the value (loss or acc) of each epoch in training set?
or the value (loss or acc) of each epoch in testing set?
I am so confused about how to arrange the Intermediate and Finally results.

[JianJuly]

Also, how to used my validation set? Or no validation set is needed?

[QuanluZhang]

@JianJuly , intermediate result is mainly used for users to check a configuration/trial's performance, like learning curve. You can report any value or dict you want. Finally result is your optimization goal. It can be seen as the reward used by tuning algorithms (e.g., tpe). how frequently to report intermediate result can be freely decided by yourself. Note that if you are using curve fitting early stop algorithm, suggest not to make intermediate result very frequent, as curve fitting is computation intensive.

how to use validation set is depending on your task decided by yourself. NNI does not have specific requirement for how to use validation set.

[JianJuly]

@JianJuly , intermediate result is mainly used for users to check a configuration/trial's performance, like learning curve. You can report any value or dict you want. Finally result is your optimization goal. It can be seen as the reward used by tuning algorithms (e.g., tpe). how frequently to report intermediate result can be freely decided by yourself. Note that if you are using curve fitting early stop algorithm, suggest not to make intermediate result very frequent, as curve fitting is computation intensive.

how to use validation set is depending on your task decided by yourself. NNI does not have specific requirement for how to use validation set.

Thanks a lot. Your suggestion of not to make intermediate result very frequent is very valuable. I'd like to report my training loss/acc as intermediate result , and testing auc/acc as final result.