John Snow Labs Spark-NLP 3.1.3: TF Hub support, new multilingual NER models for 40 languages, state-of-the-art multilingual sentence embeddings for 100+ languages, and bug fixes!

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 3.1.3! In this release, we bring notebooks to easily import models for BERT and ALBERT models from TF Hub into Spark NLP, new multilingual NER models for 40 languages with a fine-tuned XLM-RoBERTa model, and new state-of-the-art document/sentence embeddings models for English and 100+ languages!

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Support BERT models from TF Hub to Spark NLP
• Support BERT for sentence embeddings from TF Hub to Spark NLP
• Support ALBERT models from TF Hub to Spark NLP
• Welcoming new Databricks 8.4 / 8.4 ML/GPU runtimes to Spark NLP platforms

---

New Models

We have trained multilingual NER models by using the entire XTREME (40 languages) and WIKINER (8 languages).

Multilingual Named Entity Recognition:

Model	Name	Build	Lang
NerDLModel	ner_xtreme_xlm_roberta_xtreme_base	3.1.3	xx
NerDLModel	ner_xtreme_glove_840B_300	3.1.3	xx
NerDLModel	ner_wikiner_xlm_roberta_base	3.1.3	xx
NerDLModel	ner_wikiner_glove_840B_300	3.1.3	xx
NerDLModel	ner_mit_movie_simple_distilbert_base_cased	3.1.3	en
NerDLModel	ner_mit_movie_complex_distilbert_base_cased	3.1.3	en
NerDLModel	ner_mit_movie_complex_bert_base_cased	3.1.3	en

Fine-tuned XLM-RoBERTa base model by randomly masking 15% of XTREME dataset:

Model	Name	Build	Lang
XlmRoBertaEmbeddings	xlm_roberta_xtreme_base	3.1.3	xx

New Universal Sentence Encoder trained with CMLM (English & 100+ languages):

The models extend the BERT transformer architecture and that is why we use them with BertSentenceEmbeddings.

Model	Name	Build	Lang
BertSentenceEmbeddings	sent_bert_use_cmlm_en_base	3.1.3	en
BertSentenceEmbeddings	sent_bert_use_cmlm_en_large	3.1.3	en
BertSentenceEmbeddings	sent_bert_use_cmlm_multi_base	3.1.3	xx
BertSentenceEmbeddings	sent_bert_use_cmlm_multi_base_br	3.1.3	xx

---

Benchmark

We used BERT base, large, and the new Universal Sentence Encoder trained with CMLM extending the BERT transformer architecture to train ClassifierDL with News dataset:

(120k training examples - 10 Epochs - 512 max sequence - Nvidia Tesla P100)

Model	Accuracy	F1	Duration
tfhub_use	0.90	0.89	10 min
tfhub_use_lg	0.91	0.90	24 min
sent_bert_base_cased	0.92	0.90	35 min
sent_bert_large_cased	0.93	0.91	75 min
sent_bert_use_cmlm_en_base	0.934	0.91	36 min
sent_bert_use_cmlm_en_large	0.945	0.92	72 min

The complete list of all 3700+ models & pipelines in 200+ languages is available on Models Hub.

---

Bug Fixes

• Fix serialization issue in NorvigSweetingModel
• Fix the issue with BertSentenceEmbeddings model in TF v2
• Update ArrayType structure to fix Finisher failing to clean up some annotators

---

New Notebooks

Spark NLP	TF Hub Notebooks
BertEmbeddings	TF Hub in Spark NLP - BERT
BertSentenceEmbeddings	TF Hub in Spark NLP - BERT Sentence
AlbertEmbeddings	TF Hub in Spark NLP - ALBERT

---

Documentation

• HuggingFace & TF Hub to Spark NLP
• Models Hub with new models
• Spark NLP publications
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Workshop notebooks
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.1.3

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.1.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.1.3

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.1.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.1.3

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.1.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.1.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.1.3

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.1.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.1.3</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.1.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.1.3</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.1.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.1.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.1.3.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.1.3.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.1.3.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.1.3.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.1.3.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.1.3.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.1.3: TF Hub support, new multilingual NER models for 40 languages, state-of-the-art multilingual sentence embeddings for 100+ languages, and bug fixes!.