Occasional slow network connection to Buildkite and Github from Hetzner Cloud servers

[wilzbach]

Looking through the logs it sometimes took several minutes for a few git clones.
Also, I'm currently looking at this build where it takes already more than four minutes to download the distribution archive (something that typically happens in less than 2 seconds).

(https://buildkite.com/dlang/dmd/builds/2922#5fca6c19-2d09-4401-8a8c-d9587ea681f1)

Did we reach a threshold in network usage and are heavily rate-throttled?

[wilzbach]

This is definitely still a huge issue.
Just git cloning the core repos take now several minutes:

https://buildkite.com/dlang/ci/builds/592#7e4ecaf6-33e9-483e-81e1-47065faeb5d2

It uses to be a few seconds at most.

[MartinNowak]

Slow cloning times are certainly annoying but can have many different reasons. Given that the rest of the tests on that agent seem to run fine and that the slow cloning was not the cause of the test failure, I'd be careful to jump to conclusions.

[MartinNowak]

Could be congestion on the transit provider as both Buildkite (s3.amazonaws.com) and Github seem to be located at US-East.
https://www.peeringdb.com/asn/24940
https://bgp.he.net/AS24940

[MartinNowak]

This might also just be CPU bound (see build times). 8 agents are quite overcommited for a 4 vCPU KVM server. The overcommiting is done intentionally to interleave I/O wait times.