520-reproduction

Test bed to reproduce 520 errors

Requirements

Terraform

Install terraform following the relevant guide here: https://developer.hashicorp.com/terraform/install

k6

Install k6 following the relevant guide here: https://grafana.com/docs/k6/latest/set-up/install-k6/

Setup the test bed

Deploy the infra with terraform

terraform init
terraform apply -var="project_id=your_project_id"

Create the DNS record in CF

Create the DNS record in CF targeting the public ip of the LB with the proxy activated.

Run the k6 test

Generate a payload file of desired size, here 4MB

dd if=/dev/urandom of=file_4mb.bin bs=1024 count="$((4 * 1024))" status=none

Run k6 test with against the chosen host with the chosen payload

k6 run --env FILE_PATH=file_4mb.bin --env URL=https://cf-test.photoroom.com k6_test.js

Test results

k6 run --env FILE_PATH=file_4mb.bin --env URL=https://cf-test.photoroom.com k6_test.js

         /\      Grafana   /‾‾/
    /\  /  \     |\  __   /  /
   /  \/    \    | |/ /  /   ‾‾\
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/

     execution: local
        script: k6_test.js
        output: -

     scenarios: (100.00%) 1 scenario, 100 max VUs, 1m0s max duration (incl. graceful stop):
              * default: Up to 100 looping VUs for 30s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

WARN[0009] Request Failed                                error="stream error: stream ID 1; INTERNAL_ERROR; received from peer"
WARN[0012] Request Failed                                error="stream error: stream ID 7; INTERNAL_ERROR; received from peer"
WARN[0013] Request Failed                                error="stream error: stream ID 7; INTERNAL_ERROR; received from peer"
WARN[0016] Request Failed                                error="stream error: stream ID 5; INTERNAL_ERROR; received from peer"
WARN[0017] Request Failed                                error="stream error: stream ID 5; INTERNAL_ERROR; received from peer"
WARN[0021] Request Failed                                error="stream error: stream ID 11; INTERNAL_ERROR; received from peer"

     ✗ is status 200
      ↳  15% — ✓ 131 / ✗ 730

     checks...........................: 15.21% 131 out of 861
     data_received....................: 2.3 MB 72 kB/s
     data_sent........................: 3.6 GB 112 MB/s
     http_req_blocked.................: avg=3.91ms   min=334ns    med=582ns    max=58.3ms   p(90)=29.17ms  p(95)=33.62ms
     http_req_connecting..............: avg=1.28ms   min=0s       med=0s       max=12.21ms  p(90)=10.49ms  p(95)=11.14ms
     http_req_duration................: avg=1.46s    min=357.26ms med=1.53s    max=2.18s    p(90)=1.99s    p(95)=2.09s
       { expected_response:true }.....: avg=611.13ms min=357.26ms med=548.51ms max=1.15s    p(90)=980.77ms p(95)=1.08s
     http_req_failed..................: 84.78% 730 out of 861
     http_req_receiving...............: avg=5.82ms   min=45.51µs  med=174.91µs max=1s       p(90)=391.22µs p(95)=462.25µs
     http_req_sending.................: avg=91.75ms  min=62.18ms  med=91.26ms  max=144.53ms p(90)=100.33ms p(95)=104.65ms
     http_req_tls_handshaking.........: avg=2.56ms   min=0s       med=0s       max=45.13ms  p(90)=18.1ms   p(95)=21.7ms
     http_req_waiting.................: avg=1.37s    min=270.46ms med=1.43s    max=2.09s    p(90)=1.9s     p(95)=2s
     http_reqs........................: 861    26.685262/s
     ✓ { status:!200, status:!520 }...: 0      0/s
     ✓ { status:0 }...................: 6      0.18596/s
     ✓ { status:200 }.................: 131    4.060127/s
     ✓ { status:520 }.................: 724    22.439175/s
     iteration_duration...............: avg=2.48s    min=1.36s    med=2.54s    max=3.23s    p(90)=3s       p(95)=3.11s
     iterations.......................: 861    26.685262/s
     vus..............................: 2      min=2          max=100
     vus_max..........................: 100    min=100        max=100


running (0m32.3s), 000/100 VUs, 861 complete and 0 interrupted iterations
default ✓ [======================================] 000/100 VUs  30s

Run curl test

Generate a payload file of desired size, here 4MB

dd if=/dev/urandom of=file_4mb.bin bs=1024 count="$((4 * 1024))" status=none

Run curl test

paulberthaux:cf-520/ $ while true; do curl -X POST https://cf-test.photoroom.com -d @file_4mb.bin; done
{"code":200}
{"code":200}
{"code":200}
{"code":200}
{"code":200}
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
{"code":200}
{"code":200}
{"code":200}
error code: 520{"code":200}
{"code":200}
error code: 520{"code":200}
{"code":200}