1% of traffic receives 502

[davejohnston]

I have pushpin deployed in GCP on a pod.
We have a JS client that connects an opens a HTTP stream. The client first makes a CORS header request followed by the GET request. Alot of these transactions originate from behind a single NAT router.

As load increases on the pushpin pod we are seeing a number of 502 HTTP responses. Looking at the pod resources both CPU and Memory are low and not being stressed, so its unclear why we are getting the 502 for some requests.

We exec'd onto the pod where the issue was happening and used curl in a loop to make a request to the orgin server directly. We didn't seen any errors over a period of time.
We then repeated the test, this time using curl to hit http://localhost:7999, proxying via pushpin. In this case we saw occasional 502 errors, so it seems like an issue with pushpin itself.

In our production logs we see lots of these log messages.
[proxy] GET http://events.mycompany.com/stream?cluster=2 -> client-server-clusterip:80 ref=https://remotehost.com/ code=502 32 ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

We also see zurl logs
[zurl] OUT id=5c5d2ae9-503e-4c0d-a606-e44828d844c6 code=200 2
I can see these OUT messages with codes 200 and 204. But no 502s.

Could anyone help us trace this issue. What log messages should we expect to see before/after the proxy log with the 502?