HADOOP-19336: S3A: Test failures after CSE support added #7164

shameersss1 · 2024-11-15T17:38:46Z

Description of PR

Test fix and documentation changes. This commit does the following

Moved troubleshooting steps of encryption from troubleshooting_s3.md into encryption.md
Added new troubleshooting step when KMS region is not provided or doesn't match and 400 status code is returned
Skipped tests when CSE is enabled for

ITestAwsSdkWorkarounds.testNoisyLogging
ITestUploadRecovery.testCommitOperations

The reason for the same is mentioned here

For ITestUploadRecovery.testMagicWriteRecovery - It is expected to fail when CSE is enabled and pass otherwise. It helps to capture the case when this gets fixed in future

How was this patch tested?

tested in us-east-1 with (both per bucket and base s3 configuration )

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-11-15T19:02:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 17s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 20s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 30s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 24s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 50s		trunk passed
+1 💚	shadedclient	20m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 21s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 12s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	20m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 13s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		82m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/1/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 241354e79570 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `59a2ce8`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/1/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-17T07:38:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 6s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 19s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 48s		trunk passed
+1 💚	shadedclient	20m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	20m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 14s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		89m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/2/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 0f6bab47f1d2 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `18a6b9f`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/2/testReport/
Max. process+thread count	750 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-11-18T08:14:45Z

@steveloughran - Could you please review the changes

steveloughran · 2024-11-19T15:58:34Z

regarding ITestUploadRecovery we try to regenerate the failure condition "whole block uploaded then server returns 500"

that is: it is testing that the SDK will recover from such a failure by resubmitting the block again.
If this doesn't work then will CSE code be able to recover from a failure in a part upload? the implication is "no"

In which case, although this is a rare occurrence, it does surface in the wild sporadically and it seems like this going to result in a failure with this message.

Ideally, I would like this recovery to be handled. Is this possible? what do other clients do here? do you have a special "unreliable test endpoint" to generate these failures for more accurate testing?

If it can't be recovered from and the error message can be expected

test should actually expect upload failure and the error if encryption is enabled. This will highlight if/when there's a change.
troubleshooting docs need something on the error and the meaning.

hadoop-yetus · 2024-11-19T16:51:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 19s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 21s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 30s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	20m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 43s		the patch passed
+1 💚	shadedclient	20m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 8s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		82m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/3/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 302be6a3e1c1 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cffc4ae`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/3/testReport/
Max. process+thread count	699 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-11-20T10:20:58Z

@steveloughran

Retrying MPU with CSE is not supported : aws/amazon-s3-encryption-client-java#268 (comment)

Sure, i will modify the test to fail when cse is used and add the error stack in toubleshooting

hadoop-yetus · 2024-11-20T13:34:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 44s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 23s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 27s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 25s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 19s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 44s		trunk passed
-1 ❌	shadedclient	24m 45s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 21s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 10s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 15s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 40s		the patch passed
-1 ❌	shadedclient	25m 59s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 23s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 24s		ASF License check generated no output?
		91m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/4/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 3550139cb5cd 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9b7f17a`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/4/testReport/
Max. process+thread count	552 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-20T14:06:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	0m 21s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	compile	0m 8s	/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	hadoop-aws in trunk failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌	compile	0m 22s	/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.txt	hadoop-aws in trunk failed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.
-0 ⚠️	checkstyle	4m 12s	/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 22s	/branch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in trunk failed.
+1 💚	javadoc	1m 25s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 49s		trunk passed
-1 ❌	shadedclient	1m 19s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 22s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	0m 21s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌	javac	0m 21s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚	compile	0m 24s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 24s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌	mvnsite	0m 21s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javadoc	0m 21s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌	javadoc	0m 21s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.
-1 ❌	spotbugs	0m 21s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	shadedclient	7m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 20s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 21s		ASF License check generated no output?
		16m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/5/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 3fcd1a0dd366 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b4662ac`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/5/testReport/
Max. process+thread count	88 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-11-20T14:38:53Z

not sure what's up with yetus/jenkins right now

shameersss1 · 2024-11-21T04:54:18Z

Yeah, looks like some problem with yetus, let me retrigger the pipeline

hadoop-yetus · 2024-11-21T06:19:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 0s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 23s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 20s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	20m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 44s		the patch passed
+1 💚	shadedclient	20m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 13s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		82m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/6/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 30a768802f6a 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `97c8a78`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/6/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM
+1

added failure of recovery stack trace and explanation

steveloughran

Added the extra stack trace and text needed to explain that upload failure. Someone will encounter it -having a searchable section on the error and what it means will stop people filing new JIRAs (they will ask the same SO question repeatedly, but that's not our problem)

steveloughran · 2024-11-22T13:59:41Z

merged with my extra stack trace. I was writing as something I wanted you to add, then remembered I can edit a PR -and for docs that's safe.

hadoop-yetus · 2024-11-22T15:16:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 1s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 21s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 27s		trunk passed
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 48s		trunk passed
+1 💚	shadedclient	20m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 24s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 16s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	0m 14s		the patch passed
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	19m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 9s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		81m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/7/artifact/out/Dockerfile
GITHUB PR	#7164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux c100cf281e5a 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d2085cb`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/7/testReport/
Max. process+thread count	642 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7164/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

github-actions bot added trunk TOOLS AWS labels Nov 15, 2024

shameersss1 force-pushed the HADOOP-19336 branch from 9b7f17a to b4662ac Compare November 20, 2024 13:49

HADOOP-19336: S3A: Test failures after CSE support added

97c8a78

shameersss1 force-pushed the HADOOP-19336 branch from b4662ac to 97c8a78 Compare November 21, 2024 04:56

steveloughran reviewed Nov 22, 2024

View reviewed changes

Update encryption.md

d2085cb

added failure of recovery stack trace and explanation

steveloughran approved these changes Nov 22, 2024

View reviewed changes

steveloughran merged commit 31b8b4a into apache:trunk Nov 22, 2024
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19336: S3A: Test failures after CSE support added #7164

HADOOP-19336: S3A: Test failures after CSE support added #7164

shameersss1 commented Nov 15, 2024 •

edited

Loading

hadoop-yetus commented Nov 15, 2024

hadoop-yetus commented Nov 17, 2024

shameersss1 commented Nov 18, 2024

steveloughran commented Nov 19, 2024

hadoop-yetus commented Nov 19, 2024

shameersss1 commented Nov 20, 2024

hadoop-yetus commented Nov 20, 2024

hadoop-yetus commented Nov 20, 2024

steveloughran commented Nov 20, 2024

shameersss1 commented Nov 21, 2024

hadoop-yetus commented Nov 21, 2024

steveloughran left a comment

steveloughran left a comment

steveloughran commented Nov 22, 2024

hadoop-yetus commented Nov 22, 2024

HADOOP-19336: S3A: Test failures after CSE support added #7164

HADOOP-19336: S3A: Test failures after CSE support added #7164

Conversation

shameersss1 commented Nov 15, 2024 • edited Loading

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Nov 15, 2024

hadoop-yetus commented Nov 17, 2024

shameersss1 commented Nov 18, 2024

steveloughran commented Nov 19, 2024

hadoop-yetus commented Nov 19, 2024

shameersss1 commented Nov 20, 2024

hadoop-yetus commented Nov 20, 2024

hadoop-yetus commented Nov 20, 2024

steveloughran commented Nov 20, 2024

shameersss1 commented Nov 21, 2024

hadoop-yetus commented Nov 21, 2024

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran commented Nov 22, 2024

hadoop-yetus commented Nov 22, 2024

shameersss1 commented Nov 15, 2024 •

edited

Loading