Skip to content

Version 4.9.1

Compare
Choose a tag to compare
@huseinzol05 huseinzol05 released this 01 Sep 17:01
· 222 commits to master since this release
  1. Added pretrained KenLM models, trained on https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean, https://malaya.readthedocs.io/en/latest/load-kenlm.html
  2. Improved spelling correction interface, under malaya.spelling_correction.*.
  3. Improved JamSpell spelling correction interface, https://malaya.readthedocs.io/en/latest/load-spelling-correction-jamspell.html
  4. Improved speed and accuracy Probability spelling correction, https://malaya.readthedocs.io/en/latest/load-spelling-correction-probability.html
  5. Added Probability LM, probability + KenLM spelling correction, a better scoring based on sentence context, https://malaya.readthedocs.io/en/latest/load-spelling-correction-probability-lm.html
  6. Improved Spylls spelling correction interface, https://malaya.readthedocs.io/en/latest/load-spelling-correction-probability-lm.html
  7. Improved SymSpeller spelling correction interface, https://malaya.readthedocs.io/en/latest/load-spelling-correction-symspell.html
  8. Improved Transformer Encoder spelling correction interface, https://malaya.readthedocs.io/en/latest/load-spelling-correction-encoder-transformer.html
  9. Improved Seq2Seq Transformer spelling correction interface, https://malaya.readthedocs.io/en/latest/load-spelling-correction-transformer.html
  10. Added Syllable tokenizer, https://malaya.readthedocs.io/en/latest/load-tokenizer-syllable.html
  11. Added stemmer trained on noisy dataset to achieve better stemming for local language structure, https://malaya.readthedocs.io/en/latest/load-stemmer.html#Sensitive-towards-local-language-structure
  12. Improved normalizer, now able to add stemmer and add more parameters, https://malaya.readthedocs.io/en/latest/load-normalizer.html