mc admin update is extremly slow since version RELEASE.2024-01-28T16-23-14Z (server is already running latest version)

[spranta-devops]

Expected behavior

When minio server is running the latest version, then calling mc admin update <ALIAS> should return in a seconds (or a few seconds).

Here is the (sanitized) debug output with the last fast working mc version (from mcli_20240118070339.0.0_amd64.deb):

time mc admin update minio01 --debug
You are about to upgrade *MinIO Server*, please confirm [y/N]: y
mc: <DEBUG> POST /minio/admin/v3/update?updateURL= HTTP/1.1
Host: minio01.sanitized.com
User-Agent: MinIO (linux; amd64) madmin-go/2.0.0 mc/RELEASE.2024-01-18T07-03-39Z
Transfer-Encoding: chunked
Accept-Encoding: zstd,gzip
Authorization: sanitized
X-Amz-Content-Sha256: sanitized
X-Amz-Date: 20240708T074246Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 81
Accept-Ranges: bytes
Content-Type: application/json
Date: Mon, 08 Jul 2024 07:42:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: sanitized
X-Amz-Id-2: sanitized
X-Amz-Request-Id: sanitized
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

Server `minio01` already running the most recent version 2024-07-04T14:25:45Z of MinIO

real    0m0.811s
user    0m0.055s
sys     0m0.026s

Actual behavior

It needs about 2m30s to return. I had to increase the server timeout to prevent a gateway timeout error from the load balancer (HAProxy). Or use a minio node directly as alias, without using the load balancer.

The (sanitized) debug output:

time mc admin update minio01 --debug
You are about to upgrade *MinIO Server*, please confirm [y/N]: y
mc: <DEBUG> POST /minio/admin/v3/update?dry-run=false&type=2&updateURL= HTTP/1.1
Host: minio01.sanitized.com
User-Agent: MinIO (linux; amd64) madmin-go/2.0.0 mc/RELEASE.2024-01-28T16-23-14Z
Transfer-Encoding: chunked
Accept-Encoding: zstd,gzip
Authorization: sanitized
X-Amz-Content-Sha256: sanitized
X-Amz-Date: 20240709T050957Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 963
Accept-Ranges: bytes
Content-Type: application/json
Date: Tue, 09 Jul 2024 05:12:26 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: sanitized
X-Amz-Id-2: sanitized
X-Amz-Request-Id: sanitized
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

Server update request sent successfully `minio01`
┌──────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐
│ HOST                                     │ STATUS                                                             │
├──────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ minio01-node02.sanitized.com:9000 │ server is already running the latest version: 2024-07-04T14:25:45Z │
│ minio01-node01.sanitized.com:9000 │ server is already running the latest version: 2024-07-04T14:25:45Z │
│ minio01-node03.sanitized.com:9000 │ server is already running the latest version: 2024-07-04T14:25:45Z │
│ minio01-node04.sanitized.com:9000 │ server is already running the latest version: 2024-07-04T14:25:45Z │
│ minio01-node05.sanitized.com:9000 │ server is already running the latest version: 2024-07-04T14:25:45Z │
└──────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘

real    2m31.105s
user    0m0.075s
sys     0m0.038s

Steps to reproduce the behavior

Run mc admin update <ALIAS> against a server that is already running the latest version. (2024-07-04T14:25:45Z at the moment)

For testing set the ALIAS to a node and not the load balancer URL. This is easier to set up.

• The last fast version of mc is: RELEASE.2024-01-18T07-03-39Z
• The first slow version of mc is: RELEASE.2024-01-28T16-23-14Z

mc --version

mc version RELEASE.2024-07-08T20-59-24Z (commit-id=21d3ec0089a1fa297cbdc23db413012535e2ff9e)
Runtime: go1.22.5 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

Running mc on a Ubuntu VM