How to use Spark NLP in Java

[ansorre]

On home page of website: https://nlp.johnsnowlabs.com/ I read "Full Python, Scala, and Java support"

Unfortunately it's 3 days now I'm trying to use Spark NLP in Java without any success.

• I cannot find Java API (JavaDoc) of the framework.
• not event a single example in Java is available
• I do not know Scala, I do not know how to convert things like:
  val testData = spark.createDataFrame(Seq((1, "Google ..."),(2, "The Paris ..."))).toDF("id", "text")
  from Scala to Java neither searching on Google for 3 days was of any help, neither trying on my own I was able to make this conversion.

Sorry to annoy with this but is there anyone out there who is really using Spark NLP with Java? How do you do it? Are there online resources available to learn how to do it?
Thanks anyway.

[maziyarpanahi]

Hi @Antonio-Sorrentini
I put Java on the home page as I had the understanding that the APIs can be easily used in Java. I might be wrong or Java needs its own documentation like Scala and Python. So let me ping some other developers and we get to the bottom of this.

PS: your code example is pure Apache Spark to manually create a DataFrame, not really Spark NLP.

[ansorre]

Ok @maziyarpanahi thanks.

[alinapetukhova]

Hi, @Antonio-Sorrentini ! Here is example for Java pipeline with Spark-NLP. You can find lib in maven repo.

SparkSession spark = SparkSession
                .builder()
                .appName("PipelineExample")
                .config("spark.master", "local")
                .getOrCreate();

Dataset<Row> data = spark.read().format("csv")
                .option("inferSchema", "true")
                .option("header", "true")
                .option("multiLine", "true")
                .option("escape", "\"")
                .load("data.csv");

DocumentAssembler document_assembler = (DocumentAssembler) new DocumentAssembler().setInputCol("text").setOutputCol("document");

SentenceDetector sentence_detector = (SentenceDetector) ((SentenceDetector) new SentenceDetector().setInputCols(new String[] {"document"})).setOutputCol("sentence");

Tokenizer tokenizer = (Tokenizer)((Tokenizer) new Tokenizer().setInputCols(new String[] {"sentence"})).setOutputCol("token");

Normalizer normalizer = (Normalizer)((Normalizer) new Normalizer().setInputCols(new String[]{"token"})).setOutputCol("normalized");

LemmatizerModel lemmatizer = (LemmatizerModel)((LemmatizerModel) LemmatizerModel.pretrained("lemma_antbnc", "en", "public/models").setInputCols(new String[]{"normalized"})).setOutputCol("lemma");

Finisher finisher = new Finisher().setInputCols(new String[]{"document", "lemma"}).setOutputCols(new String[]{"document", "lemma"});

Pipeline pipeline = new Pipeline().setStages(new PipelineStage[]{document_assembler, sentence_detector, tokenizer, normalizer, lemmatizer, finisher});

// Fit the pipeline to training documents.
PipelineModel pipelineFit = pipeline.fit(data);
Dataset<Row> dataet = pipelineFit.transform(data);

[dkincaid]

I am trying to use a pretrained pipeline and running into an issue.

SparkSession spark = SparkSession
          .builder()
          .appName(appName)
          .master(master)
          .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1")
          .getOrCreate();

PretrainedPipeline pipeline = new PretrainedPipeline("explain_document_ml", "en",
        PretrainedPipeline$.MODULE$.apply$default$3());

List<StructField> fields = new ArrayList<StructField>();
StructField field1 = DataTypes.createStructField("id", DataTypes.IntegerType, true);
StructField field2 = DataTypes.createStructField("text", DataTypes.StringType, true);
fields.add(field1);
fields.add(field2);

StructType schema = DataTypes.createStructType(fields);

List<Row> rows = Lists.newArrayList(
          RowFactory.create(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
          RowFactory.create(2, "The Paris metro will soon enter the 21st century, ditching single-use paper tickets for rechargeable electronic cards.")
);

Dataset<Row>  testData = spark.createDataFrame(rows, schema);
Dataset<Row> annotation = pipeline.transform(testData);

I'm getting a NoClassDefFoundError that I can't quite figure out. It is getting thrown on the line creating the PretrainedPIpeline object. I tried explicitly including the scala-library jar( version 2.12.x) in my project, but that didn't fix it. Any ideas?

Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
	at com.johnsnowlabs.nlp.pretrained.PretrainedPipeline.<init>(PretrainedPipeline.scala:7)
	at com.idexx.nlp.johnsnow.poc.Runner.run(Runner.java:34)
	at com.idexx.nlp.johnsnow.poc.Runner.main(Runner.java:66)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 3 more

[dkincaid]

In case anyone else comes across this I did get my NoClassDefFound error fixed. My problem was that I was using the spark-code_2.12 version, but Spark NLP is built with _2.11 (which I should have seen originally since it's in the name of the library). Once I changed the spark lib to _2.11 it works.

[almogtavor]

You can use the Javadoc reference here https://javadoc.io/doc/com.johnsnowlabs.nlp/spark-nlp-gpu_2.11/latest/index.html#package. That's not enough to be able to completely work with the library, but that's a start.

[mwunderlich]

In case anyone else comes across this I did get my NoClassDefFound error fixed. My problem was that I was using the spark-code_2.12 version, but Spark NLP is built with _2.11 (which I should have seen originally since it's in the name of the library). Once I changed the spark lib to _2.11 it works.

@dkincaid It seems SparkNLP has now moved to Scala 2.12. Did you try again with that version? What did you do to change the spark-lib to 2.11?

I am asking because I am also getting a ClassNotFoundException: org.apache.spark.ml.PipelineModel$ using SparkNLP_2.12 version 3.3.0. Explicitly adding the Apache Spark dependency to the POM did not fix this issue.

[mwunderlich]

In case anyone else comes across this I did get my NoClassDefFound error fixed. My problem was that I was using the spark-code_2.12 version, but Spark NLP is built with _2.11 (which I should have seen originally since it's in the name of the library). Once I changed the spark lib to _2.11 it works.

@dkincaid It seems SparkNLP has now moved to Scala 2.12. Did you try again with that version? What did you do to change the spark-lib to 2.11?

I am asking because I am also getting a ClassNotFoundException: org.apache.spark.ml.PipelineModel$ using SparkNLP_2.12 version 3.3.0. Explicitly adding the Apache Spark dependency to the POM did not fix this issue.

Answering my own question: I was able to fix the ClassNotFoundException by explitly add the following dependency to my POM:

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.12</artifactId>
            <version>3.0.2</version>
        </dependency>

This is a bit strange, because the SparkNLP POM does specify the required dependency, so this should not be necessary, as I would expect it to get pulled in as a transitive dependency. I presume this is because the scope of the SparkNLP dependency is provided and therefore Spark is not packaged with SparkNLP. It makes sense, but is confusing for a first time user of SparkNLP.

[maziyarpanahi]

Spark NLP starts the support for Scala 2.12 since 3.x release. We now support both Scala _2.11 and _2.12 for Apache Spark 2.3.x, 2.4.x (Scala 2.11), and Apache Spark 3.0.x and 3.1.x (Scala 2.12)

I presume this is because the scope of the SparkNLP dependency is provided

You are absolutely correct, since the Apache Spark dependencies are provided it won't pull them nor do we ship them with our Fat JARs. It must be provided by the user, loaded already in the classpath such environments with Apache Spark (Databricks, EMR, etc.).

[seinecle]

I published this repo featuring lemmatization with Spark NLP using the Java language. It is used in a web app deployed here.