Custom vocab size mismatch

[saiful9379]

Checks

• This template is only for bug reports, usage problems go with 'Help Wanted'.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I confirm that I am using English to submit this report in order to facilitate communication.

Environment Details

Env same as defined

Steps to Reproduce

1. Used the train.py script.

✔️ Expected Behavior

I have trained the model for 17 epochs and stopped for some purpose to debug, but when i want to train model previous pretrain checkpoint, it raises a shape mismatch error. I have vocab size 134, it always show 135 shape.

❌ Actual Behavior

Train the scarch vocab file 134, but from the training checkpoint, it shows 135.

[SWivid]

the filler token <F> is also a vocab
and normally if you have not rerun the script to prepare dataset (i.e. not overwrite previous vocab.txt), will not have mismatch error

[saiful9379]

@SWivid Thank you for your reply, but I want to know where to add extra tokens when loading a pretrained checkpoint. I have changed the data loader without preparing the dataset. Could you please mention where to add extra token when loading pretrain.