Use different positional embeddings

[aflah02]

Hi
Really cool repo. Is there any way I can use different positional encodings during training? I want to essentially train several versions of tiny llama with say rope, absolute, and no positional encodings.

[zzhhjjj]

I think this will require you to modify the code a little bit.

[aflah02]

I think this will require you to modify the code a little bit.

Can you point me to where you think these changes need to be done? Should I like a new copy of https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/llama.py with different encodings? or should I be doing this elsewhere?