John Snow Labs Spark-NLP 3.2.3: New Transformers and Training documentation, Improved GraphExtraction, new Japanese models, new multilingual Transformer models, enhancements, and bug fixes

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 3.2.3! This release comes with new and completed documentation for all Transformers and Trainable annotators in Spark NLP, new Japanese NER and Embeddings models, new multilingual Transformer models, code enhancements, and bug fixes.

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Add delimiter feature to CoNLL() class to support other delimiters in CoNLL files Add delimiter feature to CoNLL class #5934
• Add support for IOB in addition to IOB2 format in GraphExtraction annotator Fix graph extraction light pipeline #6101
• Change YakeModel output type from KEYWORD to CHUNK to have more available features after the YakeModel annotator such as Chunk2Doc or ChunkEmbeddings Change KEYWORD to CHUNK output type in YakeModel #6065
• Welcoming Databricks Runtime 9.0, 9.0 ML, and 9.0 ML with GPU
• A new and completed Transformer page
  • description
  • default model's name
  • link to Models Hub
  • link to notebook on Spark NLP Workshop
  • link to Python APIs
  • link to Scala APIs
  • link to source code and unit test
  • Examples in Python and Scala for
    • Prediction
    • Training
    • Raw Embeddings
• A new and completed Training page
  • Training Datasets
  • Text Processing
  • Spell Checkers
  • Token Classification
  • Text Classification
  • External Trainable Models

---

Bug Fixes & Enhancements

• Fix the default language for XlmRoBertaSentenceEmbeddings pretrained model in Python Fix default language for XlmRoBertaSentenceEmbeddings pretrained model #6057
• Fix SentenceEmbeddings issue concatenating sentences instead of each correspondent sentence Fix Sentence Embeddings issue #6060
• Fix GraphExctraction usage in LightPipeline Fix graph extraction light pipeline #6101
• Fix compatibility issue in explain_document_ml pipeline
• Better import process for corrupted merges file in Longformer tokenizer Avoid exceptions due to corrupt merges in Longformer tokenizer #6083

---

Models and Pipelines

Spanish, Greek, Swedish, Dutch, German, French, Romanian, and Japanese

BERT Embeddings (Word and Sentence)

Model	Name	Build	Lang
BertEmbeddings	bert_base_uncased_legal	3.2.2	en
BertEmbeddings	bert_base_uncased	3.2.2	es
BertEmbeddings	bert_base_cased	3.2.2	es
BertEmbeddings	bert_base_uncased	3.2.2	el
BertEmbeddings	bert_base_cased	3.2.2	sv
BertEmbeddings	bert_base_cased	3.2.2	nl
BertSentenceEmbeddings	sent_bert_base_uncased_legal	3.2.2	en
BertSentenceEmbeddings	sent_bert_base_uncased	3.2.2	es
BertSentenceEmbeddings	sent_bert_base_cased	3.2.2	es
BertSentenceEmbeddings	sent_bert_base_uncased	3.2.2	el
BertSentenceEmbeddings	sent_bert_base_cased	3.2.2	sv
BertSentenceEmbeddings	sent_bert_base_cased	3.2.2	nl

Other multilingual models

Model	Name	Build	Lang
WordEmbeddingsModel	japanese_cc_300d	3.2.2	ja
NerDLModel	ner_ud_gsd_cc_300d	3.2.2	ja
NerDLModel	ner_ud_gsd_xlm_roberta_base	3.2.2	ja
BertForTokenClassification	bert_token_classifier_ner_ud_gsd	3.2.2	ja
BertForTokenClassification	bert_token_classifier_ner_btc	3.2.2	en
ClassifierDLModel	classifierdl_bert_sentiment	3.2.2	de
ClassifierDLModel	classifierdl_bert_sentiment	3.2.2	fr

The complete list of all 3700+ models & pipelines in 200+ languages is available on Models Hub.

---

Models Hub for the community by community

Serve Your Spark NLP Models for Free! You can host and share your Spark NLP models & pipelines publicly with everyone to reuse them with one line of code!

Models Hub is open to everyone to upload their models and pipelines, showcase their work, and share them with others.

Please visit the following page for more information: https://modelshub.johnsnowlabs.com/

---

Documentation

• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.2.3

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.2.3

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.2.3

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.2.3

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.2.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.2.3</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.2.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.2.3</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.2.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.2.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.2.3.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.2.3.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.2.3.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.2.3.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.2.3.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.2.3.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.2.3: New Transformers and Training documentation, Improved GraphExtraction, new Japanese models, new multilingual Transformer models, enhancements, and bug fixes.