araGPT2 finetuning in conversation systems

[En-J-A]

What is the maximum number for epochs to finetune the araGPT2 to conversation systems? and If I have little data, can I trained the model or it is necessary to have a large dataset?

[WissamAntoun]

No idea, but probably not more than 5 or 6 epochs. I say try it for 5 epochs and test the model at every epoch

[En-J-A]

I tried to finetune the model but the code goes to the part of (Training new model from scratch)
and then gives me the following error
I don't understand what is wrong?

WARNING:__main__:Process rank: -1, device: cuda, n_gpu: 1distributed training: False, 16-bits training: False
INFO:__main__:Training/evaluation parameters Namespace(train_data_file='./aragpt2-base/train', output_dir
='output', model_type='aubmindlab/aragpt2-base', eval_data_file='./aragpt2-base/val', model_name_or_path='aubmindlab/aragpt2-base', use_tpu=False, config_name=None, tokenizer_name=None, cache_dir=None, block_size=-1, do_train=True, do_eval=False, evaluate_during_training=True, per_gpu_train_batch_size=4, per_gpu_eval_batch_size=4, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-06, lamb_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=100000, warmup_steps=10, logging_s
teps=1000, save_steps=10000, save_total_limit=None, eval_all_checkpoints=False, no_cuda=False, overwrite_output_dir=False, overwrite_cache=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, n_gpu=1, device=device(type='cuda'))
[INFO|configuration_utils.py:530] 2021-07-07 08:49:23,496 >> loading configuration file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/config.json from cache at C:\Users/.cache\huggingface\transformers\c848bb22c5a0d744f1e3abd4c1fb45592d7e26004c12d272f15abbb677c
[INFO|configuration_utils.py:566] 2021-07-07 08:49:23,497 >> Model config GPT2Config {
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 0,
  "embd_pdrop": 0.1,
  "eos_token_id": 0,
  "gradient_checkpointing": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": null,
  "n_layer": 12,
  "n_positions": 1024,
  "resid_pdrop": 0.1,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50,
      "no_repeat_ngram_size": 3,
      "num_beams": 5,
      "repetition_penalty": 3.0,
      "top_p": 0.95
    }
  },
  "transformers_version": "4.8.2",
  "use_cache": true,
  "vocab_size": 64000
}

[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,631 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/vocab.json from cache at C:\Users/.cache\huggingface\transformers\39ed1d90869e4c124ff4dda50bb8182.404fbc3128ba88c2a81adce6919908c7a9b5682cd
[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,631 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/merges.txt from cache at C:\Users/.cache\huggingface\transformers\824410a1fcd36f179e72d22f382566af8cdde.cbf9eb42f3b5137cd5eaa99adea84e1c16cb45efd57ac9
[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,632 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/tokenizer.json from cache at C:\Users/.cache\huggingface\transformers\ec3273db6f68a482bd88e783fc4ebcb4320e65e6abf1e1b9.6ce583bd59bf6fb3ac34ead1e0fa04af22fb464906ee74d4c25bd77
[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,632 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,632 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:1717] 2021-07-07 08:49:29,632 >> loading file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/tokenizer_config.json from cache at None
[INFO|modeling_utils.py:1161] 2021-07-07 08:49:30,740 >> loading weights file https://huggingface.co/aubmindlab/aragpt2-base/resolve/main/pytorch_model.bin from cache at C:\Users/.cache\huggingface\transformers\2kbaaa4c39d355154a89a40a4b7247b6a6e750ca95avf98b03.a3beb309b2f850febd831012d0ac3e33433821a08623c868eb852c7c473
[INFO|modeling_utils.py:1345] 2021-07-07 08:49:33,502 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.

[INFO|modeling_utils.py:1353] 2021-07-07 08:49:33,502 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at aubmindlab/aragpt2-base.
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
INFO:__main__:Training new model from scratch
INFO:__main__:Training/evaluation parameters Namespace(train_data_file='./aragpt2-base/train', output_dir
='output', model_type='aubmindlab/aragpt2-base', eval_data_file='./aragpt2-base/val', model_name_or_path='aubmindlab/aragpt2-base', use_tpu=False, config_name=None, tokenizer_name=None, cache_dir=None, block_size=-1, do_train=True, do_eval=False, evaluate_during_train
ing=True, per_gpu_train_batch_size=4, per_gpu_eval_batch_size=4, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-06, lamb_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=100000, warmup_steps=10, logging_steps=1000, save_steps=10000, save_total_limit=None, eval_all_checkpoints=False, no_cuda=False, overwrite_output_dir=False, overwrite_cache=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, n_gpu=1, device=device(type='cuda'))
./aragpt2-base/train
INFO:data.dataset.language_model:Creating features from dataset file at ./aragpt2-base/train
Traceback (most recent call last):

  File "main.py", line 448, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "data\dataset\language_model.py", line 190, in 
load_and_cache_examples
    return LineByLineTextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "data\dataset\language_model.py", line 24, in 
__init__ self.examples = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size,truncation=True)["input_ids"
]
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_base.py", line 2495, in batch_e
ncode_plus
    return self._batch_encode_plus(
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\models\gpt2\tokenization_gpt2_fast.py", line 163,
in _batch_encode_plus
    return super()._batch_encode_plus(*args, **kwargs)
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_fast.py", line 377, in _batch_e
ncode_plus
    self.set_truncation_and_padding(
  File "C:\Users\miniconda3\envs\torch\lib\site-packages\transformers\tokenization_utils_fast.py", line 335, in set_trun
cation_and_padding
    self._tokenizer.enable_truncation(max_length, stride=stride, strategy=truncation_strategy.value)
OverflowError: can't convert negative int to unsigned

[WissamAntoun]

This looks like an issue with the tokenizer, which version of transformers are you using (also please tell me know which version of the other libraries are you using)

[En-J-A]

transformer --> 4.8.2
torch ---> 1.8.1
numpy --> 1.19.5

[salama4ai]

generally, there is no maximum number of epochs , every new epoch the model is enhanced more, but the mount of that enhancement reduces every new epoch, for example the first epoch the enhancement would be say 40% the second epoch would be say 15% the third would be 3% and so on , and you will find at the end after a given epoch number that the mount of the enhancement is zero or trivial and doesn't deserve the time and the computational resources consumption, as i know increasing number of epochs will never hurt your algorithm
you can train the model with little data, but the issue that the model performance will not be stunning ,

[En-J-A]

Ok. thanks.