Replies: 4 comments
-
|
Please elaborate on the use case for Spark beyond what is already supported. I believe there are already users who are using Spark in a Nextflow process, and you can perform SQL queries with the |
Beta Was this translation helpful? Give feedback.
-
|
Can you help me understand what is already supported? I cannot find any official documentation. Is there more than this repository which "contains a reusable set of Nextflow subworkflows and processes which create transient Apache Spark clusters on any infrastructure where Nextflow runs"? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
That repo is a great example of what's already supported, basically using Spark in a process like any other application. The maintainer of that repo (Konrad) would be a good person to talk to if you want to learn more. I believe he gave a talk at the 2022 Nextflow Summit about it. |
Beta Was this translation helpful? Give feedback.
-
|
https://summit.nextflow.io/2022/program/oct-13-large-scale-image-processing-with-nextflow/ |
Beta Was this translation helpful? Give feedback.
-
New feature
Add support for Apache Spark to allow developers to connect legacy applications with distributed analytics engines.
Beta Was this translation helpful? Give feedback.
All reactions