Too many requests (429) on local cache server with high parallelism

[norman-zon]

Describe the bug

When running terragrunt run-all init -upgrade with TERRAGRUNT_PROVIDER_CACHE=1 with high parallelism leads to 429 errors from the local cache server:

Error: Could not retrieve providers for locking
 
OpenTofu failed to fetch the requested providers for linux_amd64 in order
to calculate their checksums: some providers could not be installed:
- registry.opentofu.org/hashicorp/google: could not query provider registry
for registry.opentofu.org/hashicorp/google: the request failed after 2
attempts, please try again later: 429 Too Many Requests returned from
127.0.0.1:41797

Steps To Reproduce

Run terragrunt run-all init -upgrade with TERRAGRUNT_PROVIDER_CACHE=1 in a setup with a lot (ca >20) terragrunt modules without limiting --terragrunt-parallelism .

Expected behavior

The local cache server accepts all connections

Versions

• Terragrunt version: 0.59.6
• OpenTofu version: 1.7.0
• Environment details (Ubuntu 20.04, Windows 10, etc.): Github Action Runner, based on Ubuntu 22.04

Additional context

Possible solutions:

Either increase the limit for the cache server, or add a TERRAGRUNT_PROVIDER_CACHE_REQ_LIMIT variable to let the user decide.