Tokentisation

[kritisingh24]

In the [paper] (https://www.cfilt.iitb.ac.in/iitb_parallel/lrec2018_iitbparallel.pdf) it is mentioned, that the data is tokenised by using Indic_NLP language for Hindi and Moses for English, but this is not the case with the given example code, are the results still reproducible if we avoid this step?

[dipteshkanojia]

The results shall be reproducible with the help of the methodology mentioned in the paper. The example code just helps is provided to help a beginner in the area start with the task. Always recommended to use the method described in the paper.