Training spacy model stucking in 99%

[Rafa7046]

Description

So i am training a NER model using transformers and my gpu, the problem happens in the training when the bar that keeps tracks of the progress reachs 99%, it takes 5 minutes to reach 99% and then take many more to go from 99 to 100 maybe even an hour,this is the way that was supposed to work?

How to reproduce the behaviour

Your Environment

• Operating System: Windows server 2019, but i also managed to reproduce in linux ubuntu 20.04 and windows 11
• Python Version Used: 3.11
• spaCy Version Used: 3.7.2

config.txt

[svlandeg]

Hi! To try and figure out what may be going on, can I ask you some more questions?

• Does it get stuck in the 99% for every row it prints, or only in the beginning or end of an epoch?
• When you monitor your system resources during the training loop, do you see hints of maybe the GPU being overcommitted?
• How big is your training data set? How big your evaluation data set?
• What kind of transformer model are you using, on what kind of GPU?
• Are you sure the GPU is used? (just to double check you're not accidentally running this on CPU - the output log should say so)

[Rafa7046]

Hi, thanks for the reply!

• For all rows it get stuck at 99
• It doesn't seem to be overcommitting the GPU
• Train has 6000 data points and 1500 for the evaluation
• The transformers detail can be found in the config.txt i left as a file in the original post, the GPU is a RTX 4060 TI
• Yes i am sure is using the GPU i can see the log in the beginning of the process and i also see the GPU being used in the resource monitor

A print from the training

[svlandeg]

Train has 6000 data points and 1500 for the evaluation

That's pretty reasonable - my main guess was (and still is) that the delay is caused by running the pipeline on the full eval set, to print out the F/P/R scores after the model has gone through a training iteration.

Maybe just to test this hypothesis, could you try a run with the same training dataset, but a much reduced eval dataset, containing only a handful of data points? I'd expect the training to still take the same amount of time, but the progress to go from 99% to 100% much more quickly. Just to verify whether my intuition is correct or not, so we can then look into where/how to resolve the original issue.

[Rafa7046]

Hello, sorry for the delay. I tried this, i used 6780 for training and only 34 for eval and still gets stuck on the 99%