Split long sentences into arrays of capped token length

[jenghub]

Hi there! I want to take very long sentences in my corpus, and split them at a max length (say 512 tokens). Are there any utilities to help perform this in spark NLP? Would the best way to be to use tokenizer(), do some list manipulation, and then re-join elements in the capped token arrays to reconstruct the split sentences? Thanks!

[maziyarpanahi]

Hi @jenghub

We have 2 sentence splitter annotators: SentenceDetector which is rule-based and SentenceDetectorDL which is a trainable and DL-based annotator. (SentenceDetectorModel for prediction)

Both of these annotators have a parameter called splitLength to split the sentence at that length into 2 or more sentences:

• https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/sbd/pragmatic/SentenceDetector.html#splitLength:org.apache.spark.ml.param.IntParam
• https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/sentence_detector_dl/SentenceDetectorDLModel.html#splitLength:org.apache.spark.ml.param.IntParam

Hope this helps :)

[jenghub]

Wow, totally missed that. Thanks so much @maziyarpanahi !

[jenghub]

@maziyarpanahi and just to clarify. This is token or word length rather than character length?

[maziyarpanahi]

it splits everything by empty space and then counts it if I am not mistaken. (the actual tokenizer comes after this annotator so it simply takes the white spaces as words and that is considered the number of words)