Harbor is throwing 503s #21250
Comments
veerendra2
changed the title
|
maybe something wrong with the redis and postgres. postgres DB restart 88 minute ago, what is the deployment type of three postgresql instance? |
|
@stonezdj We use https://github.com/zalando/postgres-operator to manage postgresql $ k get postgresql
NAME TEAM VERSION PODS VOLUME CPU-REQUEST MEMORY-REQUEST AGE STATUS
harbor-postgres-db harbor 15 3 2Gi 100m 1024Mi 130d Running
$ k get sts
NAME READY AGE
harbor-postgres-db 3/3 19h
harbor-trivy 3/3 130d
redis-node 3/3 130d
$ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
harbor ClusterIP 10.0.134.156 <none> 80/TCP 130d
harbor-core ClusterIP 10.0.198.244 <none> 80/TCP,8001/TCP 130d
harbor-exporter ClusterIP 10.0.255.72 <none> 8001/TCP 49d
harbor-jobservice ClusterIP 10.0.233.25 <none> 80/TCP,8001/TCP 130d
harbor-portal ClusterIP 10.0.129.86 <none> 80/TCP 130d
harbor-postgres-db ClusterIP 10.0.218.184 <none> 5432/TCP 130d
harbor-postgres-db-config ClusterIP None <none> <none> 130d
harbor-postgres-db-repl ClusterIP 10.0.5.126 <none> 5432/TCP 130d
harbor-registry ClusterIP 10.0.234.173 <none> 5000/TCP,8080/TCP,8001/TCP 130d
harbor-trivy ClusterIP 10.0.134.5 <none> 8080/TCP 130d
patroni-metrics ClusterIP 10.0.31.14 <none> 9547/TCP 30d
postgres-exporter ClusterIP 10.0.197.179 <none> 9187/TCP 30d
redis ClusterIP 10.0.186.250 <none> 6379/TCP,26379/TCP 130d
redis-headless ClusterIP None <none> 6379/TCP,26379/TCP 130d
redis-metrics ClusterIP 10.0.250.114 <none> 9121/TCP 130dWe updated postgresql sidecar container before, that's why there were some restarts. Please let me know any further details are needed EDIT Attaching jager traces on an endpoint
|
|
I increased the EDIT
|
|
Update I even check a sha256 layer path really exists in my storage account, indeed the sha256 layer for the image is exists in storage account
Searched same sha256 layer in Azure Storage Account Explorer
And also there are lot of ClientErrors/Failed transactions in Azure storage account insights(Anyways these ClientErrors are exists for long time)
|
|
It seems mainly the upstream (harbor-core) is resetting the connection. By the way increased number replicas from 3 to 5 Still, same, getting 503 |
|
What is the output in the harbor-core log? Harbor core doesn't throw 503 error in the program, this error is usually thrown by front end components. |
reasonerjt
added
the
help wanted
I already attached harbor-core debug logs here Upstream (harbor-core) resetting the connection, that's why We had to add retry for virtualservice like below to fix |
|
Hello @veerendra2 Can you add some additional logs from the istio-proxy (for all the core pods), actual container logs instead of the klogs output? I am curious if some of your traffic is routed towards the blackhole cluster. Additinally are you using REGISTRY_ONLY outbound traffic policy instead of ALLOW_ANY? If yes, are you properly configuring the service-entry, the destination rules and the gateway? Most of the 503 I investigated on Istio were solely because the traffic was routed to the blackhole cluster due to a misconfigured virtualservice, gateway or destinationrule. |
Please find the attachment for logs of
If this the case, this should happen all the times, but in my case it is happening once in a while. After adding retries, there is lot better client experience and almost no 503s from clients(i.e docker login in Github actions and docker pulls)
It should be EDIT apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
labels:
app: harbor-core
kustomize.toolkit.fluxcd.io/name: harbor
kustomize.toolkit.fluxcd.io/namespace: flux-system
name: harbor-core
namespace: harbor
spec:
host: harbor-core.harbor.svc.cluster.local
trafficPolicy:
connectionPool:
http:
idleTimeout: 50s
loadBalancer:
simple: LEAST_REQUESTAnd similar for |
|
And also there are still lot of client errors shown in storage account insights |
We get 503 errors once in a while for
/v2/*endpoint and we have to re-run the github pipeline to make it work.Below is the setup, deployed harbor via helm
Attaching screenshots
Harbor metrics grafana dashboard
istio-gateway logs for harbor which HTTP response 503
Steps to reproduce the problem:
Versions:
Please specify the versions of following systems.
1.16.0[v2.12.0](https://github.com/goharbor/harbor/releases/tag/v2.12.0)v1.30.3Additional context:
The text was updated successfully, but these errors were encountered: