New features and optimizations

Added public-facing official Dockerfile for NeMo-Aligner.
PPO: memory optimization to help avoid OOM in the actor when sending training data to the critic.
PPO: it is now possible to use a custom end string in sampling_params.end_strings that is different from <extra_id_1>.
SFT: added support for custom validation metrics based on model generations.
Added the ability to do multi-epoch (cfg.max_epochs > 1) training for reward models, DPO, PPO, and SFT
SFT/SteerLM: added LoRA tuning as an option besides full fine-tuning, only attention_qkv layer is supported

Breaking changes

We have changed the shuffle logic in the data sampler to support multi-epoch training, so training runs using identical parameters
will not give the same results anymore because the shuffle logic has changed (specifically the seed value is modified slightly per epoch).
If you run CI/regression type tests, then be warned that the test may break due to this shuffle change.

Fixed a potential issue when the base model's model.data.data_prefix config is a list and is about to be overridden with
a dictionary from the training configuration.
exp_manager.max_time_per_run is now respected, the trainers will save and run validation before exiting if we've reached the time limit.
Fixed crash in PPO when using a separate reward model server (i.e., with combine_rm_and_critic_server=False).
Fixed crash when LR scheduler is not specified

docker pull nvcr.io/nvidia/nemo:24.01.framework

To get access:

Sign up to get free and immediate access to NVIDIA NeMo Framework container. If you don’t have an NVIDIA NGC account, you will be prompted to sign up for an account before proceeding.
If you don’t have an NVIDIA NGC API key, sign into NVIDIA NGC, selecting organization/team: ea-bignlp/ga-participants and click Generate API key. Save this key for the next step. Else, skip this step.
On your machine, docker login to nvcr.io using

docker login nvcr.io
Username: $oauthtoken
Password: <Your Saved NGC API Key>