Help me to execute OLAP queries with EMR Spark cluster

[rayala-nag]

We are trying OLAP queries for one of our use case. I followed JanusGraph documentation suggested here. I'm able to execute OLAP queries with local spark mode. But want to use EMR Spark cluster to run my OLAP queries.

Any one can help me with steps? I couldn't found relavant answer to solve my issue.

[hadoopmarc]

This blog is outdated, but it still demonstrated the basics of accessing spark-yarn from JanusGraph:
http://yaaics.blogspot.com/2017/07/
Come back if you can show some stack traces!

[rayala-nag]

@hadoopmarc I'm not clear how to proceed with spark EMR cluster. If you see JanusGraph documentation, they covered setup with spark local mode/spark standalone mode. I didn't found any resource to explain the steps to execute OLAP queries with spark EMR cluster. If you share or help me with this that would be great.

[hadoopmarc]

I do not use or know AWS EMR myself. Reading from https://medium.com/nerd-for-tech/spark-remote-job-submission-to-yarn-running-on-aws-emr-5d6ffd8cb373 :

Copy Hadoop config files yarn-site.xml, core-site.xml, mapred-site.xml from /etc/hadoop/conf and copy spark config file hive-site.xml from /etc/spark/conf.

The hadoop and spark config files should be part of the classpath of your Tinkerpop app (e.g. gremlin console). The earlier link shows how to achieve that. You do not need any spark or hadoop binaries (jars) from EMR, because these are already part of the JanusGraph distribution. If the hadoop/yarn version of JanusGraph does not match that of EMR, you are out of luck.

[rayala-nag]

@hadoopmarc If you look at below SparkGraphComputer configuration what could be the Spark master URL incase of we are using EMR cluster as a option. It would be different configuration can be use for EMR Spark cluster. Please let me know your thoughts on this.

If possible share the steps if you have any to configure JanusGraph with EMR cluster. That would be great help for me.

SparkGraphComputer Configuration

spark.master=spark://127.0.0.1:7077
spark.executor.memory=1g
spark.executor.extraClassPath=/opt/lib/janusgraph/*
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator