Warning for maximum sequence length when running FSDP Llama2 example

[amanshanbhag]

In awsome-distributed-training/3.test_cases/10.FSDP, when running sbatch 1.distributed-training.sbatch ( 1.distributed-training.sbatch), a bunch of warnings that look like pop up:

1: Token indices sequence length is longer than the specified maximum sequence length for this model (2522 > 2048). Running this sequence through the model will result in indexing errors

How to reproduce:

No changes were made to any of the training python scripts. The only changes made to the 1.distributed-training.sbatch file were to change from Llama2-7B to Llama2-13B. Everything else was kept the same. Just run everything as is as per the instructions in the workshop.

There's some discussion on altering max_length, or making some adjustments to the Tokenizer in this issue. This could be helpful in fixing the warning.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.