Remote server failed to handle the request, will retry in a moment needs to be better handled

[belimawr]

• Relates Fleet-Server abruptly closes connection when limit is reached instead of returning TooManyRequests (429) fleet-server#4200

When trying to communicate with Fleet-Server (e.g: during enrol), Elastic-Agent might log the error

Remote server failed to handle the request, will retry in a moment

However this can be the result of 3 different HTTP status codes:

elastic-agent/internal/pkg/fleetapi/enroll_cmd.go

Lines 35 to 40 in 54932dc

	// temporaryServerErrorCodes defines status codes that allow clients to retry their request.
	var temporaryServerErrorCodes = map[int]struct{}{
	http.StatusBadGateway: {},
	http.StatusServiceUnavailable: {},
	http.StatusGatewayTimeout: {},
	}

Bad Gateway and Gateway Timeout are likely connectivity problems, however Service Unavailable can be a real problem with Fleet-Server.

We need to be more clear in the logs which error has happened and likely better handle them individually. E.g: Bad Gateway might be a configuration issue, while Gateway Timeout is a connectivity issue. It might not make sense to retry Bad Gateway, while Gateway Timeout should be retried with an exponential backoff.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

We should just return the exact HTTP status and any error message instead of abstracting it.

This made diagnosing elastic/fleet-server#4200 much harder than it needs to be.