Can't use basic auth to API server https://[vm-domain]:6443 on v1.21.3+k3s1

[byunru]

Environmental Info:
K3s Version: v1.21.3+k3s1

Node(s) CPU architecture, OS, and Version:
Linux dfo-ubunto8 5.8.0-1039-azure #42~20.04.1-Ubuntu SMP Thu Jul 15 14:11:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
installing v1.17.0+k3s.1, I can load https://[vm-domain]:6443 outside of vm, get correct result.
I haven't try versions in between v1.17.0+k3s.1 and v1.21.3+k3s1

Steps To Reproduce:

• Installed K3s:
• curl -sfL https://get.k3s.io | sh -

Expected behavior:
https://[vm-domain]:6443 can be load outside of vm, with prompt token input dialog, after enter the token, will display the list of api url

Actual behavior:
load https://[vm-domain]:6443 from outside of vm, get error message as below:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}

Additional context / logs:
------------------------------------------------ k3s install v1.21.3+k3s1 with no error --------------------------------------------
[INFO] Finding release for channel stable
[INFO] Using v1.21.3+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.3+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.3+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
------------------------------------------------ service k3s status [ v1.21.3+k3s1]-------------------------------------------
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-08-11 17:20:16 UTC; 5min ago
Docs: https://k3s.io
Process: 2357 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 2359 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 2360 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 2361 (k3s-server)
Tasks: 94
Memory: 1.3G
CGroup: /system.slice/k3s.service
├─2361 /usr/local/bin/k3s server
├─2444 containerd
├─3226 /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7f60f9e2fd5314a973b9204aae502f42eb67477024c85dd627660f4a4f05f6d3 -address /run/k3s/containerd/containerd.sock
├─3247 /pause
├─3270 /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 580d65f49e863dda5252ce1462f10672466add189593fa902e7e6401ca74d830 -address /run/k3s/containerd/containerd.sock
├─3292 /pause
├─3397 /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 663e870cff796989d1904f56f4459ddd14c57fe705afd82a7ea11d2890dbc79e -address /run/k3s/containerd/containerd.sock
├─3430 /pause
├─3592 local-path-provisioner start --config /etc/config/config.json
├─3677 /metrics-server
├─3837 /coredns -conf /etc/coredns/Corefile
├─4705 /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin/containerd-shim-runc-v2 -namespace k8s.io -id a64d1457a9adc3b0c4ce9fc5be6ee704ee2059653345db275fd6b5eee576c4b4 -address /run/k3s/containerd/containerd.sock
├─4725 /pause
├─4748 /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin/containerd-shim-runc-v2 -namespace k8s.io -id df2d42834f2ec4fe77107d84b55eea3a10afd203e9e55d69b72c3a72a44e591b -address /run/k3s/containerd/containerd.sock
├─4770 /pause
├─4963 /bin/sh /usr/bin/entry
├─5006 /bin/sh /usr/bin/entry
└─5105 traefik traefik --global.checknewversion --global.sendanonymoususage --entryPoints.traefik.address=:9000/tcp --entryPoints.web.address=:8000/tcp --entryPoints.websecure.address=:8443/tcp --api.dashboard=true --ping=true --providers.kubernetescrd --providers.kubernetesingress --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik --entrypoints.websecure.http.tls=true

Aug 11 17:21:51 ubunto8 k3s[2361]: time="2021-08-11T17:21:51.848879233Z" level=info msg="Cluster-Http-Server 2021/08/11 17:21:51 http: TLS handshake error from 50.64.28.166:56311: remote error: tls: unknown certificate authority"
Aug 11 17:21:54 ubunto8 k3s[2361]: I0811 17:21:54.428691 2361 trace.go:205] Trace[356108114]: "GuaranteedUpdate etcd3" type:*core.Endpoints (11-Aug-2021 17:21:53.250) (total time: 1178ms):
Aug 11 17:21:54 ubunto8 k3s[2361]: Trace[356108114]: ---"Transaction committed" 1177ms (17:21:00.428)
Aug 11 17:21:54 ubunto8 k3s[2361]: Trace[356108114]: [1.178267878s] [1.178267878s] END
Aug 11 17:21:54 ubunto8 k3s[2361]: I0811 17:21:54.428818 2361 trace.go:205] Trace[895343909]: "Update" url:/api/v1/namespaces/kube-system/endpoints/rancher.io-local-path,user-agent:local-path-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.42.0.2,accept:application/json, /,protocol:HTTP/1.1 (11-Aug-2021 17:21:53.250) (total time: 1178ms):
Aug 11 17:21:54 ubunto8 k3s[2361]: Trace[895343909]: ---"Object stored in database" 1178ms (17:21:00.428)
Aug 11 17:21:54 ubunto8 k3s[2361]: Trace[895343909]: [1.178636503s] [1.178636503s] END
Aug 11 17:22:18 ubunto8 k3s[2361]: time="2021-08-11T17:22:18.067471604Z" level=info msg="Cluster-Http-Server 2021/08/11 17:22:18 http: TLS handshake error from 50.64.28.166:54339: remote error: tls: unknown certificate"
Aug 11 17:22:22 ubunto8 k3s[2361]: time="2021-08-11T17:22:22.085748116Z" level=info msg="Cluster-Http-Server 2021/08/11 17:22:22 http: TLS handshake error from 50.64.28.166:55713: remote error: tls: unknown certificate"
Aug 11 17:22:22 ubunto8 k3s[2361]: time="2021-08-11T17:22:22.216398603Z" level=info msg="Cluster-Http-Server 2021/08/11 17:22:22 http: TLS handshake error from 50.64.28.166:52351: EOF"

============================ correct senario on v1.17.0+k3s.1 ============================
------------------------------------------------ k3s install v1.17.0+k3s.1 with no error --------------------------------------------
~$ curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.17.0+k3s.1 sh -
[INFO] Finding release for channel v1.17.0+k3s.1
[INFO] Using v1.17.0+k3s.1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.17.0+k3s.1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.17.0+k3s.1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
------------------------------------------------ service k3s status [v1.17.0+k3s.1]-------------------------------------------
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-08-11 17:55:57 UTC; 5min ago
Docs: https://k3s.io
Process: 11407 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 11410 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 11411 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 11412 (k3s-server)
Tasks: 89
Memory: 999.1M
CGroup: /system.slice/k3s.service
├─11412 /usr/local/bin/k3s server
├─11707 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
├─12332 /var/lib/rancher/k3s/data/c67259f824ace42864ead7655dbe15a5b736ad91050498224cd42f777416d8dd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 71433835a43157448c5bc170c28542f7df0b7c69256aeb0f95d11ee0dc644371 -address /run/k3s/containerd/containerd.sock
├─12354 /pause
├─12441 /var/lib/rancher/k3s/data/c67259f824ace42864ead7655dbe15a5b736ad91050498224cd42f777416d8dd/bin/containerd-shim-runc-v2 -namespace k8s.io -id b3dcbdaf3c68a58ca58f253b2189313a71932d431ba7c44919905506356e3dd3 -address /run/k3s/containerd/containerd.sock
├─12467 /pause
├─12469 /var/lib/rancher/k3s/data/c67259f824ace42864ead7655dbe15a5b736ad91050498224cd42f777416d8dd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7452963c0ca31a54be1902fb6d5dc30903ae3da72c31a7f536e18df84c9d0ba6 -address /run/k3s/containerd/containerd.sock
├─12520 /pause
├─12653 /metrics-server
├─12769 /coredns -conf /etc/coredns/Corefile
├─12827 local-path-provisioner start --config /etc/config/config.json
├─13228 /var/lib/rancher/k3s/data/c67259f824ace42864ead7655dbe15a5b736ad91050498224cd42f777416d8dd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3dda22731eea430608a196e11bf80d135795711b35f1fe831ea884d2b0831f27 -address /run/k3s/containerd/containerd.sock
├─13253 /pause
├─13326 /var/lib/rancher/k3s/data/c67259f824ace42864ead7655dbe15a5b736ad91050498224cd42f777416d8dd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6d4420777c192ff010a6ccdc223c6f52bf29abb05c15ac6ee797671ab688ce49 -address /run/k3s/containerd/containerd.sock
├─13350 /pause
├─13475 /bin/sh /usr/bin/entry
├─13569 /bin/sh /usr/bin/entry
└─13630 /traefik --configfile=/config/traefik.toml

Aug 11 17:57:04 -ubunto8 k3s[11412]: I0811 17:57:04.292617 11412 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "config" (UniqueName: "kubernetes.io/configmap/81ed8ccd-ccae-40c3-868d-b51a8e36eafa-config") pod "traefik-6787cddb4b-t6d4s" (UID: "81ed8ccd-ccae-40c3-868d-b51a8e36eafa")
Aug 11 17:57:04 -ubunto8 k3s[11412]: I0811 17:57:04.493114 11412 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-lcsb7" (UniqueName: "kubernetes.io/secret/e80c6416-9b25-4168-b735-f785a4d17ec6-default-token-lcsb7") pod "svclb-traefik-wp6gf" (UID: "e80c6416-9b25-4168-b735-f785a4d17ec6")
Aug 11 17:57:04 -ubunto8 k3s[11412]: I0811 17:57:04.795694 11412 reconciler.go:183] operationExecutor.UnmountVolume started for volume "helm-traefik-token-gz4pf" (UniqueName: "kubernetes.io/secret/d408a4cd-ab6e-4efa-9da5-3e412b03504c-helm-traefik-token-gz4pf") pod "d408a4cd-ab6e-4efa-9da5-3e412b03504c" (UID: "d408a4cd-ab6e-4efa-9da5-3e412b03504c")
Aug 11 17:57:04 -ubunto8 k3s[11412]: I0811 17:57:04.815886 11412 operation_generator.go:713] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d408a4cd-ab6e-4efa-9da5-3e412b03504c-helm-traefik-token-gz4pf" (OuterVolumeSpecName: "helm-traefik-token-gz4pf") pod "d408a4cd-ab6e-4efa-9da5-3e412b03504c" (UID: "d408a4cd-ab6e-4efa-9da5-3e412b03504c"). InnerVolumeSpecName "helm-traefik-token-gz4pf". PluginName "kubernetes.io/secret", VolumeGidValue ""
Aug 11 17:57:04 -ubunto8 k3s[11412]: I0811 17:57:04.896033 11412 reconciler.go:303] Volume detached for volume "helm-traefik-token-gz4pf" (UniqueName: "kubernetes.io/secret/d408a4cd-ab6e-4efa-9da5-3e412b03504c-helm-traefik-token-gz4pf") on node "-ubunto8" DevicePath ""
Aug 11 17:57:05 -ubunto8 k3s[11412]: W0811 17:57:05.688204 11412 pod_container_deletor.go:75] Container "942a560077ca0717d076ae5c51378232a646aaaf3c2ec5fd5902ef17074ee422" not found in pod's containers
Aug 11 17:57:56 -ubunto8 k3s[11412]: I0811 17:57:56.783622 11412 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Aug 11 17:58:56 -ubunto8 k3s[11412]: I0811 17:58:56.786091 11412 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Aug 11 17:59:56 -ubunto8 k3s[11412]: I0811 17:59:56.795353 11412 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Aug 11 18:00:56 -ubunto8 k3s[11412]: I0811 18:00:56.798393 11412 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io

Backporting

• No Needs backporting to older releases

[byunru]

install "v1.17.0+k3s.1", the behaviour is correct as:

[brandond]

Support for Basic authentication (the thing that you're using with 1.17) has been removed from upstream Kubernetes since the 1.19 release. Note that this is not just deprecated or disabled, but deleted from the codebase: kubernetes/kubernetes#89069