[VL] Add Scala 2.13 support #6326

Preetesh2110 · 2024-07-03T09:52:00Z

What changes were proposed in this pull request?

Add Scala 2.13 support

(Fixes: #6316)

How was this patch tested?

Build locally

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

github-actions · 2024-07-03T09:52:16Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2024-07-03T09:52:30Z

Run Gluten Clickhouse CI

github-actions · 2024-07-03T13:19:14Z

Run Gluten Clickhouse CI

surnaik · 2024-07-04T07:10:52Z

@zhztheplayer Could you please review this PR? Thanks!

LuciferYang · 2024-07-04T07:27:59Z

Prior to Spark 4.0, Scala 2.13 was not the default Scala version, and I personally do not recommend supporting it for Spark 3.x.

surnaik · 2024-07-08T06:39:42Z

Thanks @LuciferYang, I think Scala 2.13 support is already added for CH backend, plan was to extend to VL backend as well, since the build fails when you enable -Pbackends-velox -Pscala-2.13, that was the plan, does that make sense.

FYI, the support for Scala 2.13 was added here already - #6311

cc: @zhztheplayer

surnaik · 2024-07-08T06:40:08Z

@Preetesh2110 please take a look at failing CI and maybe also add a CI option to build with Scala 2.13

LuciferYang · 2024-07-09T11:02:14Z

Thanks @LuciferYang, I think Scala 2.13 support is already added for CH backend, plan was to extend to VL backend as well, since the build fails when you enable -Pbackends-velox -Pscala-2.13, that was the plan, does that make sense.

FYI, the support for Scala 2.13 was added here already - #6311

cc: @zhztheplayer

Got

github-actions · 2024-07-10T05:14:28Z

Run Gluten Clickhouse CI

Preetesh2110 · 2024-07-10T07:07:01Z

thanks @surnaik. fixed the scala Style issues.

zhztheplayer · 2024-07-11T01:31:11Z

Do we need to have a test for Scala 2.13 (with Spark 3.5 probably) ? https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_docker.yml

github-actions · 2024-07-11T05:57:06Z

Run Gluten Clickhouse CI

Preetesh2110 · 2024-07-11T09:55:56Z

Thanks for the feedback. I have added a workflow @zhztheplayer.

zhztheplayer · 2024-07-12T00:46:07Z

Run Gluten Clickhouse CI

zhztheplayer · 2024-07-12T00:50:26Z

.github/workflows/velox_docker.yml

+      - name: Build and Run unit test for Spark 3.5.1 with Scala 2.13 (other tests)
+        run: |
+          cd $GITHUB_WORKSPACE/
+          export SPARK_SCALA_VERSION=2.13
+          $MVN_CMD clean install -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-ut -DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" -DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags && \
+          $MVN_CMD test -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Piceberg -Pdelta -DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest


This could make job run-spark-test-spark35 much slower.

If so, probably we should add a job run-spark-test-spark35-scala213. cc @zhouyuan @PHILO-HE

Done. Thanks @zhztheplayer

github-actions · 2024-07-12T02:38:08Z

Run Gluten Clickhouse CI

github-actions · 2024-07-12T05:10:30Z

Run Gluten Clickhouse CI

github-actions · 2024-07-12T09:45:46Z

Run Gluten Clickhouse CI

github-actions · 2024-07-12T09:46:56Z

Run Gluten Clickhouse CI

github-actions · 2024-07-12T13:57:54Z

Run Gluten Clickhouse CI

github-actions · 2024-08-06T06:25:59Z

Run Gluten Clickhouse CI

Preetesh2110 · 2024-08-06T06:28:41Z

@Preetesh2110

Thank you for keeping working on this!

BTW, in case you may wait for CI to be triggered, you can open a PR in your own repo to run CI if you want.

Hey @zhztheplayer thanks for your message, how can I trigger the CIs, I have created a PR in my own repo Preetesh2110#1.

github-actions · 2024-08-06T07:10:49Z

Run Gluten Clickhouse CI

github-actions · 2024-08-06T08:28:58Z

Run Gluten Clickhouse CI

github-actions · 2024-08-06T08:49:53Z

Run Gluten Clickhouse CI

github-actions · 2024-08-06T09:00:16Z

Run Gluten Clickhouse CI

github-actions · 2024-08-06T18:57:12Z

Run Gluten Clickhouse CI

zhztheplayer · 2024-08-07T01:11:24Z

@Preetesh2110
Thank you for keeping working on this!
BTW, in case you may wait for CI to be triggered, you can open a PR in your own repo to run CI if you want.

Hey @zhztheplayer thanks for your message, how can I trigger the CIs, I have created a PR in my own repo Preetesh2110#1.

You can sync gluten's main branch to your own main branch, then open a PR targeting to your own main branch. The CI should be directly runnable.

Anyway you can still update here if it's OK for you to wait for CI to be triggered.

Once you have one PR merged you will not get blocked in your subsequent PRs.

github-actions · 2024-08-07T04:02:23Z

Run Gluten Clickhouse CI

github-actions · 2024-08-07T08:15:18Z

Run Gluten Clickhouse CI

github-actions · 2024-08-08T07:49:15Z

Run Gluten Clickhouse CI

github-actions · 2024-08-08T08:20:46Z

Run Gluten Clickhouse CI

zhouyuan · 2024-08-09T08:37:08Z

@Preetesh2110 FYI, below mvn command can fix the style issues so this will not stop CI
https://github.com/apache/incubator-gluten/blob/main/docs/developers/NewToGluten.md?plain=1#L111

thanks, -yuan

surnaik

LGTM - Non binding

surnaik · 2024-08-09T09:58:06Z

@zhztheplayer please take a look, thanks!
cc: @weiting-chen

zhztheplayer

+1 overall. I haven't try with the feature yet but assume CI should cover the changes.

Thank your for the effort @Preetesh2110

.scalafmt.conf

* [VL] Add Scala 2.13 support * Fix scalaStyle issues * Fix Scala Style issues * Add Spark 3.5.1 and Scala 2.13 test in workflow * Add run-spark-test-spark35-scala213 job * Add Spark 3.5.1 and Scala 2.13 test in workflow * Fix tests failures * Fix tests failures * ScalaStyle fix * Fix SoftAffinitySuite * Fix ArrowUtil error * Fix backend-velox scala issues * Fix ColumnarArrowEvalPythonExec issues * Fix ColumnarArrowEvalPythonExec issues * Fix TestOperator.scala for style issues * Fix TestOperator.scala for style issues * Fix issues in DeltaRewriteTransformerRules.scala * DeltaRewriteTransformerRules fix * Fix style issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues --------- Co-authored-by: Hongze Zhang <[email protected]>

* [GLUTEN-6612] Fix ParquetFileFormat issue caused by the setting of local property isNativeApplicable (#6627) * [CORE] Fix schema mismatch between ReadRelNode and LocalFilesNode (#6746) Co-authored-by: 蒋添 <[email protected]> * [UT] Test input_file_name, input_file_block_start & input_file_block_length when scan falls back (#6318) * [VL] Fix E function fallback issue (#6397) * [VL] Add Scala 2.13 support (#6326) * [VL] Add Scala 2.13 support * Fix scalaStyle issues * Fix Scala Style issues * Add Spark 3.5.1 and Scala 2.13 test in workflow * Add run-spark-test-spark35-scala213 job * Add Spark 3.5.1 and Scala 2.13 test in workflow * Fix tests failures * Fix tests failures * ScalaStyle fix * Fix SoftAffinitySuite * Fix ArrowUtil error * Fix backend-velox scala issues * Fix ColumnarArrowEvalPythonExec issues * Fix ColumnarArrowEvalPythonExec issues * Fix TestOperator.scala for style issues * Fix TestOperator.scala for style issues * Fix issues in DeltaRewriteTransformerRules.scala * DeltaRewriteTransformerRules fix * Fix style issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues --------- Co-authored-by: Hongze Zhang <[email protected]> * [VL] Fix Alinux3 arrow build issue (#6363) * update velox docker and port PR #6363 for get_velox.sh update --------- Co-authored-by: PHILO-HE <[email protected]> Co-authored-by: jiangjiangtian <[email protected]> Co-authored-by: 蒋添 <[email protected]> Co-authored-by: 高阳阳 <[email protected]> Co-authored-by: Preetesh2110 <[email protected]> Co-authored-by: Hongze Zhang <[email protected]> Co-authored-by: Joey <[email protected]>

[VL] Add Scala 2.13 support

cab90d7

Fix scalaStyle issues

03bbe04

Fix Scala Style issues

418f0e5

Add Spark 3.5.1 and Scala 2.13 test in workflow

f236d57

zhztheplayer reviewed Jul 12, 2024

View reviewed changes

Add run-spark-test-spark35-scala213 job

c6eaa1a

Merge branch 'main' into VLScala-2.13

0df95dd

Preetesh2110 added 2 commits July 12, 2024 09:41

Add Spark 3.5.1 and Scala 2.13 test in workflow

a0b0059

Fix tests failures

3d40195

Fix tests failures

b0aac94

Merge branch 'main' into VLScala-2.13

bb5e0cd

DeltaRewriteTransformerRules fix

7edcb0b

Fix style issues

017c2b1

Fix issues

b4467b0

Fix issues

958d71e

Merge branch 'main' into VLScala-2.13

929bcc7

Fix issues

6db7f17

Fix issues

da3f371

Fix issues

4f69e81

Fix issues

2235752

github-actions bot added the CLICKHOUSE label Aug 8, 2024

Fix issues

6fe4db3

surnaik approved these changes Aug 9, 2024

View reviewed changes

zhztheplayer approved these changes Aug 9, 2024

View reviewed changes

.scalafmt.conf Show resolved Hide resolved

weiting-chen merged commit 51d0a37 into apache:main Aug 10, 2024
43 checks passed

[VL] Add Scala 2.13 support #6326

[VL] Add Scala 2.13 support #6326

Conversation

Preetesh2110 commented Jul 3, 2024

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

surnaik commented Jul 4, 2024

LuciferYang commented Jul 4, 2024

surnaik commented Jul 8, 2024

surnaik commented Jul 8, 2024

LuciferYang commented Jul 9, 2024

github-actions bot commented Jul 10, 2024

Preetesh2110 commented Jul 10, 2024

zhztheplayer commented Jul 11, 2024

github-actions bot commented Jul 11, 2024

Preetesh2110 commented Jul 11, 2024

zhztheplayer commented Jul 12, 2024

zhztheplayer Jul 12, 2024

Choose a reason for hiding this comment

Preetesh2110 Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Aug 6, 2024

Preetesh2110 commented Aug 6, 2024 • edited Loading

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

zhztheplayer commented Aug 7, 2024

github-actions bot commented Aug 7, 2024

github-actions bot commented Aug 7, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

zhouyuan commented Aug 9, 2024

surnaik left a comment

Choose a reason for hiding this comment

surnaik commented Aug 9, 2024

zhztheplayer left a comment • edited Loading

Choose a reason for hiding this comment

Preetesh2110 Jul 12, 2024 •

edited

Loading

Preetesh2110 commented Aug 6, 2024 •

edited

Loading

zhztheplayer left a comment •

edited

Loading