cannot reproduce the test result

[yimeng0701]

Thanks for your great work!

I have difficulty in reproducing the bleu score on the test set for the BERT2BERT model, which should be 5.58 according to your paper:

what I have done is:

1. I downloaded your pretrained BERT2BERT model from huggingface and then did the prediction on the test set. I also tried to tune the top_k, top_p and other args during generation, according to your instruction. The highest bleu score I could get on the test set is 0.91:

2. I also tried to train the model from scratch, using the train-bert2bert.ipynb file, however, the highest bleu score I could get is less than 2.

As you can see, it is far from 5.58. So could you please share with us the prediction code, as well as the values of all args, hyper-parameters (e.g. top_k, top_p, temperature, length_penalty...) that could reproduce your test result?

Thanks a lot!