Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1257851 Move to using snowjdbc thin jar instead of the fat jar #733

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

SNOW-1257851 Move to using snowjdbc thin jar instead of the fat jar #733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b42bdfc
Resolve dependency and license.
sfc-gh-alhuang Apr 9, 2024
30a5daf
Resolve undeclared dependencies and avoid import shading dependencies.
sfc-gh-alhuang Apr 9, 2024
17e6284
Add shading dependencies.
sfc-gh-alhuang Apr 9, 2024
dd95695
Modify poms for build & test.
sfc-gh-alhuang Apr 10, 2024
1abb182
Modify poms for build & test.
sfc-gh-alhuang Apr 12, 2024
5c710fd
Modify poms for fips.
sfc-gh-alhuang Apr 15, 2024
c08c9e7
merge with master
sfc-gh-alhuang Apr 15, 2024
090e362
Fix jar test.
sfc-gh-alhuang Apr 15, 2024
737825d
Fix unshaded test.
sfc-gh-alhuang Apr 18, 2024
8b00fb7
Fix unshaded test.
sfc-gh-alhuang Apr 19, 2024
92a6eab
Fix gcp test.
sfc-gh-alhuang Apr 22, 2024
a96463c
Fix gcp test.
sfc-gh-alhuang Apr 22, 2024
f92d666
Fix gcp test.
sfc-gh-alhuang Apr 23, 2024
861be1e
Fix gcp test....................
sfc-gh-alhuang Apr 23, 2024
768db23
Fix snyk check
sfc-gh-alhuang Apr 24, 2024
fc797b4
Merge branch 'master' into alhuang-jdbc-thin-jar
sfc-gh-alhuang Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion e2e-jar-test/core/src/main/java/net/snowflake/IngestTestUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,11 @@ private Properties loadProperties() throws IOException {

private Connection getConnection()
throws IOException, ClassNotFoundException, SQLException, NoSuchAlgorithmException, InvalidKeySpecException {
Class.forName("net.snowflake.client.jdbc.SnowflakeDriver");
try {
Class.forName("net.snowflake.client.jdbc.SnowflakeDriver");
} catch(ClassNotFoundException e) {
Class.forName("net.snowflake.ingest.internal.net.snowflake.client.jdbc.SnowflakeDriver");
}

Properties loadedProps = loadProperties();

Expand Down
16 changes: 10 additions & 6 deletions e2e-jar-test/fips/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<bouncycastle.bcfips.version>1.0.2.4</bouncycastle.bcfips.version>
<bouncycastle.bcpkixfips.version>1.0.5</bouncycastle.bcpkixfips.version>
</properties>

<dependencies>
Expand All @@ -27,10 +29,6 @@
<groupId>net.snowflake</groupId>
<artifactId>snowflake-ingest-sdk</artifactId>
<exclusions>
<exclusion>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk18on</artifactId>
Expand All @@ -42,8 +40,14 @@
</exclusions>
</dependency>
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc-fips</artifactId>
<groupId>org.bouncycastle</groupId>
<artifactId>bc-fips</artifactId>
<version>${bouncycastle.bcfips.version}</version>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-fips</artifactId>
<version>${bouncycastle.bcpkixfips.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
Expand Down
182 changes: 172 additions & 10 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@

<!-- Set our Language Level to Java 8 -->
<properties>
<apache.httpclient.version>4.5.14</apache.httpclient.version>
<apache.httpcomponents.version>4.4.16</apache.httpcomponents.version>
<aws.version>1.12.706</aws.version>
<bouncycastle.version>1.78.1</bouncycastle.version>
<codehaus.version>1.9.13</codehaus.version>
<commonscodec.version>1.15</commonscodec.version>
Expand All @@ -44,8 +47,10 @@
<commonslang3.version>3.14.0</commonslang3.version>
<commonslogging.version>1.2</commonslogging.version>
<commonstext.version>1.10.0</commonstext.version>
<fasterxml.version>2.14.0</fasterxml.version>
<guava.version>32.0.1-jre</guava.version>
<fasterxml.version>2.15.3</fasterxml.version>
<google.httpclient.version>1.43.3</google.httpclient.version>
<gson.version>2.10.1</gson.version>
<guava.version>32.1.1-jre</guava.version>
<hadoop.version>3.3.6</hadoop.version>
<jacoco.skip.instrument>true</jacoco.skip.instrument>
<jacoco.version>0.8.5</jacoco.version>
Expand All @@ -61,11 +66,12 @@
<objenesis.version>3.1</objenesis.version>
<parquet.version>1.13.1</parquet.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<protobuf.version>3.19.6</protobuf.version>
<protobuf.version>3.23.2</protobuf.version>
<shadeBase>net.snowflake.ingest.internal</shadeBase>
<slf4j.version>1.7.36</slf4j.version>
<snappy.version>1.1.10.4</snappy.version>
<snowjdbc.version>3.14.5</snowjdbc.version>
<threetenbp.version>1.6.9</threetenbp.version>
<yetus.version>0.13.0</yetus.version>
</properties>

Expand All @@ -78,11 +84,31 @@
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to pull all these and not get them transitively via jdbc-thin?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For com.amazonaws:aws-java-sdk-core, I got Used undeclared dependencies found error from maven-dependency-plugin if it's not explicitly specified in dependencies.

Copy link
Contributor Author

@sfc-gh-alhuang sfc-gh-alhuang May 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For other dependencies like com.google.code.gson, Failed while enforcing releasability. error is raised by maven-enforcer-plugin if it's not specified in dependencies.

Rule 3: org.apache.maven.enforcer.rules.dependency.DependencyConvergence failed with message:
[ERROR] Failed while enforcing releasability.
[ERROR] 
[ERROR] Dependency convergence error for com.google.code.gson:gson:jar:2.8.9 paths to dependency are:
[ERROR] +-net.snowflake:snowflake-ingest-sdk:jar:2.1.1-SNAPSHOT
[ERROR]   +-net.snowflake:snowflake-jdbc-thin:jar:3.14.5:compile
[ERROR]     +-com.google.cloud:google-cloud-core:jar:2.21.0:compile
[ERROR]       +-com.google.protobuf:protobuf-java-util:jar:3.23.2:compile
[ERROR]         +-com.google.code.gson:gson:jar:2.8.9:compile
[ERROR] and
[ERROR] +-net.snowflake:snowflake-ingest-sdk:jar:2.1.1-SNAPSHOT
[ERROR]   +-net.snowflake:snowflake-jdbc-thin:jar:3.14.5:compile
[ERROR]     +-com.google.cloud:google-cloud-core:jar:2.21.0:compile
[ERROR]       +-com.google.http-client:google-http-client-gson:jar:1.43.3:compile
[ERROR]         +-com.google.code.gson:gson:jar:2.10.1:compile
[ERROR] and
[ERROR] +-net.snowflake:snowflake-ingest-sdk:jar:2.1.1-SNAPSHOT
[ERROR]   +-net.snowflake:snowflake-jdbc-thin:jar:3.14.5:compile
[ERROR]     +-com.google.cloud:google-cloud-storage:jar:2.22.6:compile
[ERROR]       +-com.google.code.gson:gson:jar:2.10.1:compile

<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>${aws.version}</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>${gson.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>${google.httpclient.version}</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-gson</artifactId>
<version>${google.httpclient.version}</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
Expand Down Expand Up @@ -130,8 +156,30 @@
</dependency>
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<artifactId>snowflake-jdbc-thin</artifactId>
<version>${snowjdbc.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.grpc</groupId>
<artifactId>grpc-context</artifactId>
</exclusion>
<exclusion>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
Expand Down Expand Up @@ -161,6 +209,10 @@
<groupId>com.github.pjfanning</groupId>
<artifactId>jersey-json</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.re2j</groupId>
<artifactId>re2j</artifactId>
</exclusion>
<exclusion>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
Expand Down Expand Up @@ -255,6 +307,17 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${apache.httpclient.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>${apache.httpcomponents.version}</version>
</dependency>

<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-column</artifactId>
Expand Down Expand Up @@ -320,6 +383,11 @@
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.threeten</groupId>
<artifactId>threetenbp</artifactId>
<version>${threetenbp.version}</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
Expand Down Expand Up @@ -393,10 +461,18 @@
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
</dependency>

<!-- jwt token for key pair authentication with GS -->
<dependency>
Expand All @@ -407,6 +483,10 @@
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/io.dropwizard.metrics/metrics-core -->
<dependency>
Expand Down Expand Up @@ -439,7 +519,7 @@
<!-- Snowflake JDBC used to connect to the service-->
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<artifactId>snowflake-jdbc-thin</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
Expand All @@ -449,6 +529,14 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop -->
<dependency>
<groupId>org.apache.parquet</groupId>
Expand All @@ -475,6 +563,11 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
Expand Down Expand Up @@ -591,7 +684,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>3.0.0-M5</version>
<version>3.2.5</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand Down Expand Up @@ -817,10 +910,15 @@
|Apache-2.0
|Apache License, Version 2.0
|Apache 2.0
|Apache License V2.0</licenseMerge>
|Apache License V2.0
|Apache 2</licenseMerge>
<licenseMerge>BSD 2-Clause License
|The BSD License</licenseMerge>
<licenseMerge>The MIT License|MIT License</licenseMerge>
<licenseMerge>3-Clause BSD License
|BSD-3-Clause
|BSD New license</licenseMerge>
<licenseMerge>The MIT License|MIT License|MIT license</licenseMerge>
<licenseMerge>The Go license|Go License</licenseMerge>
</licenseMerges>
</configuration>
<executions>
Expand Down Expand Up @@ -971,12 +1069,55 @@
<configuration>
<artifactSet>
<excludes>
<exclude>net.snowflake:snowflake-jdbc</exclude>
<exclude>org.slf4j:slf4j-api</exclude>
<exclude>com.github.luben:zstd-jni</exclude>
</excludes>
</artifactSet>
<relocations>
<relocation>
<pattern>io.perfmark</pattern>
<shadedPattern>${shadeBase}.io.perfmark</shadedPattern>
</relocation>
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${shadeBase}.io.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>io.opencensus</pattern>
<shadedPattern>${shadeBase}.io.opencensus</shadedPattern>
</relocation>
<relocation>
<pattern>org.conscrypt</pattern>
<shadedPattern>${shadeBase}.org.conscrypt</shadedPattern>
</relocation>
<relocation>
<pattern>com.microsoft</pattern>
<shadedPattern>${shadeBase}.com.microsoft</shadedPattern>
</relocation>
<relocation>
<pattern>com.yammer</pattern>
<shadedPattern>${shadeBase}.com.yammer</shadedPattern>
</relocation>
<relocation>
<pattern>org.jsoup</pattern>
<shadedPattern>${shadeBase}.org.jsoup</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda</pattern>
<shadedPattern>${shadeBase}.org.joda</shadedPattern>
</relocation>
<relocation>
<pattern>org.threeten</pattern>
<shadedPattern>${shadeBase}.org.threeten</shadedPattern>
</relocation>
<relocation>
<pattern>javax.servlet</pattern>
<shadedPattern>${shadeBase}.javax.servlet</shadedPattern>
</relocation>
<relocation>
<pattern>android.annotation</pattern>
<shadedPattern>${shadeBase}.android.annotation</shadedPattern>
</relocation>
<relocation>
<pattern>com.nimbusds</pattern>
<shadedPattern>${shadeBase}.com.nimbusds</shadedPattern>
Expand Down Expand Up @@ -1075,6 +1216,22 @@
<pattern>io.airlift.compress</pattern>
<shadedPattern>${shadeBase}.io.airlift.compress</shadedPattern>
</relocation>
<relocation>
<pattern>net.snowflake.client</pattern>
<shadedPattern>${shadeBase}.net.snowflake.client</shadedPattern>
</relocation>
<relocation>
<pattern>com.snowflake.client</pattern>
<shadedPattern>${shadeBase}.com.snowflake.client</shadedPattern>
</relocation>
<relocation>
<pattern>com.amazonaws</pattern>
<shadedPattern>${shadeBase}.com.amazonaws</shadedPattern>
</relocation>
<relocation>
<pattern>software.amazon.ion</pattern>
<shadedPattern>${shadeBase}.software.amazon.ion</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
Expand All @@ -1093,7 +1250,9 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>google/protobuf/**/*.proto</exclude>
<exclude>google/**/**/*.proto</exclude>
<exclude>grpc/**/**/*.proto</exclude>
<exclude>opencensus/proto/**/v1/*.proto</exclude>
</excludes>
</filter>
<filter>
Expand Down Expand Up @@ -1234,6 +1393,9 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<classesDirectory>${project.build.OutputDirectory}</classesDirectory>
</configuration>
<executions>
<execution>
<goals>
Expand Down
6 changes: 0 additions & 6 deletions public_pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,6 @@
</scm>

<dependencies>
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<version>3.14.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
Expand Down
Loading
Loading