[bug] upload multiple files from disk with multipart upload fail

[TingDaoK]

When upload multiple files using multipart upload, the upload failed with the following error:

[ec2-user@ip-172-31-14-35 files_dir]$ /home/ec2-user/fast_ebs/aws-crt-s3-benchmarks/runners/s3-benchrunner-rust/target/release/s3-benchrunner-rust sdk-rust-tm /home/ec2-user/fast_ebs/aws-crt-s3-benchmarks/workloads/upload-5GiB-10x.run.json test-bucket-dengket us-west-2 100 --disable-directory
thread 'main' panicked at src/main.rs:49:17:
failed uploading: upload/5GiB-10x/06

Caused by:
    0: child operation failed
    1: dispatch failure
    2: timeout
    3: client error (Connect)
    4: HTTP connect timeout occurred after 3.1s
    5: timed out
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Reproduce steps:

It happens very frequently when I run the benchmark from a c5n.18xlarge instance, may happen to other type as well, but let's start with what I have tested.

• Get c5n.18xlarge instance
• Get benchmark runner https://github.com/awslabs/aws-crt-s3-benchmarks
• Follow benchmark runner guidance to build the rust runner
• Run the upload-5GiB-10x workload.

Further info

I did have some trace level logs around 500MB
https://drive.corp.amazon.com/documents/dengket@/better_trace

[aajtodd]

Is this on an al2023 image? We had other customer reports of connect issues and it turned out al2023 doesn't enable DNS caching by default and once they fixed that their connect issues went away.

[TingDaoK]

yeah, it's al2023

[ec2-user@ip-172-31-14-35 aws-crt-s3-benchmarks]$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20241121"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"