[GNMT v2/Tensorflow] Loss not decreasing when training custom dataset

[abbyDC]

I just wanted to ask the following to help me train a custom model which allows me to translate <src_lang> to english. I have an issue where the loss ranges from 17-200 for a single epoch. It goes up and down drastically. I'm not sure what else I need to tweak.

Steps I've done:

1. Edited wmt16_en_de.sh to preprocess my custom data
2. Edited nmt.py to reflect the src and tgt files.
3. FP32 training on 1 GPU

Questions:

1. What other files/scripts do I need to change for training?
2. Other ways to evaluate instead of sacrebleu? Since it uses wmt files which does not include the language I'm trying to translate.

[mwawrzos]

Hello!

There are plenty of potential reasons. It might be too high learning rate value, a problem with data preprocessing, and many other reasons.

I suggest looking for some article explaining, how to deal with such problems. Following article seems fine to me: https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607

Many articles begin with reducing the problem to the simplest example. For instance, reducing the dataset to just a few examples and checking if the model is able to overfit. If the simplest example works, other elements can be verified.

Can you try to follow this guide (or any other you find helpful)? If you will face a problem in some steps, it may be easier to help, knowing, what already works.

[abbyDC]

Thanks for the tips! I have done preprocessing on the data already and tweaked the learning rate value as well as other hyperparameters. There's not much difference though. I haven't changed anything with the core architecture of the model so I assumed it would work with other datasets as well.

Hello!

There are plenty of potential reasons. It might be too high learning rate value, a problem with data preprocessing, and many other reasons.

I suggest looking for some article explaining, how to deal with such problems. Following article seems fine to me: https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607

Many articles begin with reducing the problem to the simplest example. For instance, reducing the dataset to just a few examples and checking if the model is able to overfit. If the simplest example works, other elements can be verified.

Can you try to follow this guide (or any other you find helpful)? If you will face a problem in some steps, it may be easier to help, knowing, what already works.

[mwawrzos]

@abbyDC Have you tried to use some guide as I suggested before? Have it helped you to find any issue? Can you check the preprocessed data, for example, if all original data exists in the created dataset?

[abbyDC]

@mwawrzos Yup double checked them and there seems to be no problem with the dataset itself. Also tried doing batch inference after training and got okay results despite the loss being like that