What's the largest PPO model size and context length that have been trained successfully with this library? Can you also share some performance metrics (i.e. GPU count, training time) if possible?

[okuchaiev]

What's the largest PPO model size and context length that have been trained successfully with this library? Can you also share some performance metrics (i.e. GPU count, training time) if possible?

Originally posted by @panyi121 in #70 (comment)

[okuchaiev]

Pasting response from @HeyyyyyyG below:

Hi, we did PPO on Llama-70B model with 4k context length. In terms of GPU count, we used 32x8 GPUs for the actor and 8x8 GPUs for the critic. Try our NV-Llama2-70B-RLHF model on NVIDIA AI Foundation for free.