Should we use the new schedule-free optimizer?

[MischaPanch]

This optimizer is making some waves in the ML community. It removes the need for carefully tuning the LR schedule and generally seems to do a very good job across various modalities. We should explore to which extent it also helps us in RL. This means running some experiments and reporting results in this issue.

Somehow related to #1114. One could solve both issues in one go, as both involve the same kind of experimentation

[MischaPanch]

@arnaujc91 if you want, this is also something to take a look at together with #1114