OverflowError: can't convert negative int to unsigned (GPT2TokenizerFast)

[En-J-A]

I used the model aubmindlab/aragpt2-base and GPT2TokenizerFast in my code but I got an error:

  File "main.py", line 448, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "data\dataset\language_model.py", line 190, in 
load_and_cache_examples
    return LineByLineTextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "data\dataset\language_model.py", line 24, in 
__init__ self.examples = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size,truncation=True)["input_ids"]
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_base.py", line 2495, in batch_e
ncode_plus
    return self._batch_encode_plus(
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\models\gpt2\tokenization_gpt2_fast.py", line 163,
in _batch_encode_plus
    return super()._batch_encode_plus(*args, **kwargs)
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_fast.py", line 377, in _batch_e
ncode_plus
    self.set_truncation_and_padding(
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_fast.py", line 335, in set_trun
cation_and_padding
    self._tokenizer.enable_truncation(max_length, stride=stride, strategy=truncation_strategy.value)
OverflowError: can't convert negative int to unsigned

libraries' versions used

transformers==4.9.2
boto3==1.18.31
colorama==0.4.4
gpt-2-simple==0.7.1
ipdb==0.13.9
nltk==3.2.5
numpy==1.19.5
pip-tools==6.2.0
PyArabic==0.6.14
spacy==2.2.4
tokenizers==0.10.3
torch @ https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
torchvision @ https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp37-cp37m-linux_x86_64.whl1
tqdm==4.62.0
urllib3==1.25.11

please, how can I solve this issue?

[WissamAntoun]

can you try using it with GPT2Tokenizer and set use_fast=False when loading it?

[En-J-A]

thank you.