Why is there such a big gap between pull artifacts from the same project and different projects

[microyahoo]

Why is there such a big gap between pull artifacts from the same project and different projects? Just due to db lock?

~/go/src/github.com/goharbor/perf (main ✗) cat outputs.500/pull-artifacts-from-same-project.summary.json| jq      
{
  "timestamp": 1706025980701,
  "description": "pull-artifacts-from-same-project",
  "avg": "110.8s",
  "min": "1.24s",
  "med": "104.15s",
  "max": "260.32s",
  "p90": "208.89s",
  "p95": "233.16s",
  "successRate": "100%",
  "iterationsRate": "3.79/s"
}
~/go/src/github.com/goharbor/perf (main ✗) cat outputs.500/pull-artifacts-from-different-projects.summary.json| jq
{
  "timestamp": 1706025715365,
  "description": "pull-artifacts-from-different-projects",
  "avg": "7.19s",
  "min": "1.04s",
  "med": "6.12s",
  "max": "18.96s",
  "p90": "14.34s",
  "p95": "15.34s",
  "successRate": "100%",
  "iterationsRate": "3.58/s"
}

[wy65701436]

are those artifact you mentioned in the same projects are in the same repository? And please share more details, like the harbor version and how did you execute the performance script.

I will transfer this as a performance issue, thanks.

[microyahoo]

Thanks for your quick response, I am not sure whether this issue has related with #19902.
harbor version: v2.10.0

1. apply the env variables

export HARBOR_ROOT=/mnt/cephfs
export HARBOR_REPORT=true
export HARBOR_VUS=500
export HARBOR_URL=https://user:password@url

2. then run the go run mage.go run pull-artifacts-from-different-projects and go run mage.go run pull-artifacts-from-same-project
   pull-artifacts-from-different-projects
   pull-artifacts-from-same-project

[ianseyer]

We are seeing this behavior lead to a more than 10x slowdown when running a lot of concurrent pulls for the same project (avg time goes from 6s/req to 100s/req) when using harbor-perf's built in tests.

A key note is that pull-artifacts-from-different-projects pulls a random artifact from a random project whereas pull-artifacts-from-same-projects pulls the same artifact for all VUs.

This smells like a bug somewhere? The only write operation I could imagine stacking up would be a write to the metadata about "last pulled time" on that particular artifact