Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump cdap and hadoop version #175

Merged
merged 1 commit into from
Dec 5, 2024
Merged

Bump cdap and hadoop version #175

merged 1 commit into from
Dec 5, 2024

Conversation

psainics
Copy link
Contributor

@psainics psainics commented Dec 2, 2024 •
edited
Loading

Bump cdap and hadoop version ( Unit Test Fixes )

Bump up CDAP (6.11.0-SNAPSHOT) and Hadoop (3.3.6) version for error management.

Unit Tests Fixed

  • Modified HttpSourceETLTest.java
  • Modified HttpStreamingSourceETLTest.java
  • Modified HTTPSinkTest.java

Other changes

  1. Exclusion slf4j-reload4j
<exclusion>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-reload4j</artifactId>
</exclusion>
  • Caused error on unit tests
    Screenshot 2024-12-02 at 4 23 36 PM
  1. Adding EXPLORE_ENABLED
  • This constant was missing in the new version.

Sanity Check on new build

image
image
image
image

@psainics psainics self-assigned this Dec 2, 2024
@psainics
Copy link
Contributor Author

psainics commented Dec 2, 2024

@itsankit-google i am not confident on the exclusion can you please confirm i have done it correctly.

itsankit-google
itsankit-google reviewed Dec 2, 2024
View reviewed changes
src/test/java/io/cdap/plugin/http/etl/HttpStreamingSourceETLTest.java Outdated

@ClassRule
public static final TestConfiguration CONFIG =
new TestConfiguration(Constants.Explore.EXPLORE_ENABLED, false,
Constants.AppFabric.SPARK_COMPAT, Compat.SPARK_COMPAT);
new TestConfiguration(EXPLORE_ENABLED, false, Constants.AppFabric.SPARK_COMPAT, Compat.SPARK_COMPAT);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try removing this config altogether?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HttpStreamingSourceETLTest test keep falling, i cannot be sure if removing this is correct !
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.lang.NoClassDefFoundError usually means dependency conflicts needs to be resolved at pom level.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bump to spark-streaming and jackson version fixes this error !

image

itsankit-google
itsankit-google reviewed Dec 2, 2024
View reviewed changes
Copy link
Member

@itsankit-google itsankit-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add build.yml & build-report.yml files?

ref: https://github.com/data-integrations/salesforce/tree/develop/.github/workflows

@psainics psainics mentioned this pull request Dec 2, 2024
Merged
@psainics psainics added the build label Dec 2, 2024
@psainics psainics added build and removed build labels Dec 2, 2024
itsankit-google
itsankit-google reviewed Dec 2, 2024
View reviewed changes
pom.xml Outdated
<httpcomponents.version>4.5.9</httpcomponents.version>
<hydrator.version>2.10.0-SNAPSHOT</hydrator.version>
<hydrator.version>2.10.0</hydrator.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump it to 2.13.0-SNAPSHOT for consistency

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumped 👍

@psainics psainics force-pushed the bump/cdap-spark branch 3 times, most recently from d6a8aba to f86effa Compare December 4, 2024 10:32
@psainics
Copy link
Contributor Author

psainics commented Dec 4, 2024

The unit tests failing here are also failing in develop branch, we can have a separate PR addressing broken unit tests, and merge this one as a bump up so other developers can work with error management task.

@itsankit-google
Copy link
Member

The unit tests failing here are also failing in develop branch, we can have a separate PR addressing broken unit tests, and merge this one as a bump up so other developers can work with error management task.

Build with unit tests is a required step to pass when we bump any dependencies. We need to make sure it is green before proceeding with any other change.

@psainics
Copy link
Contributor Author

psainics commented Dec 5, 2024

🥳 Unit Tests Fixed

@psainics psainics merged commit 5a3f8ab into data-integrations:develop Dec 5, 2024
4 checks passed
@psainics psainics deleted the bump/cdap-spark branch December 5, 2024 13:42
@psainics psainics mentioned this pull request Dec 5, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@itsankit-google itsankit-google itsankit-google approved these changes

Assignees

@psainics psainics

Labels
build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@psainics @itsankit-google