agent start fail

[namehehe]

command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --server https://xx:6443 --token K10xxx::server:xxx" sh -s -

logs:

Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.253817  184981 server.go:874] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255212  184981 config.go:192] "Starting service config controller"
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255309  184981 shared_informer.go:313] Waiting for caches to sync for service config
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255430  184981 config.go:101] "Starting endpoint slice config controller"
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255473  184981 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255554  184981 config.go:319] "Starting node config controller"
Dec 01 10:02:06 ecs1503728529 k3s[184981]: I1201 10:02:06.255593  184981 shared_informer.go:313] Waiting for caches to sync for node config
Dec 01 10:02:10 ecs1503728529 k3s[184981]: E1201 10:02:10.810563  184981 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ecs1503728529\" not found"
Dec 01 10:02:12 ecs1503728529 k3s[184981]: W1201 10:02:12.123219  184981 transport.go:356] Unable to cancel request for *otelhttp.Transport
Dec 01 10:02:12 ecs1503728529 k3s[184981]: E1201 10:02:12.123537  184981 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ecs1503728529?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="1.6s"
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717257  184981 request.go:1116] Unexpected error when reading response body: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: W1201 10:02:15.717425  184981 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717414  184981 kubelet_node_status.go:96] "Unable to register node with API server" err="Post \"https://127.0.0.1:6444/api/v1/nodes\": http2: client connection lost" node="ecs1503728529"
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717449  184981 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ecs1503728529?timeout=10s\": http2: client connection lost" interval="3.2s"
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717513  184981 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: W1201 10:02:15.717550  184981 reflector.go:470] k8s.io/[email protected]/tools/cache/reflector.go:232: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717570  184981 request.go:1116] Unexpected error when reading response body: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: W1201 10:02:15.717415  184981 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.CSIDriver: Get "https://127.0.0.1:6444/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: W1201 10:02:15.717604  184981 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.RuntimeClass: Get "https://127.0.0.1:6444/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717660  184981 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://127.0.0.1:6444/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717673  184981 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://127.0.0.1:6444/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: W1201 10:02:15.717726  184981 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717770  184981 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Service: failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: http2: client connection lost
Dec 01 10:02:15 ecs1503728529 k3s[184981]: E1201 10:02:15.717705  184981 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://127.0.0.1:6444/api/v1/namespaces/default/events\": http2: client connection lost" event="&Event{ObjectMeta:{ecs1503728529.180d05542dfe4079  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ecs1503728529,UID:ecs1503728529,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ecs1503728529,},FirstTimestamp:2024-12-01 10:01:30.714538105 +0000 UTC m=+3.484556487,LastTimestamp:2024-12-01 10:01:30.714538105 +0000 UTC m=+3.484556487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ecs1503728529,}"
Dec 01 10:02:15 ecs1503728529 k3s[184981]: I1201 10:02:15.919731  184981 kubelet_node_status.go:73] "Attempting to register node" node="ecs1503728529"
Dec 01 10:02:20 ecs1503728529 k3s[184981]: E1201 10:02:20.811525  184981 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ecs1503728529\" not found"
Dec 01 10:02:28 ecs1503728529 k3s[184981]: W1201 10:02:28.918141  184981 transport.go:356] Unable to cancel request for *otelhttp.Transport
Dec 01 10:02:28 ecs1503728529 k3s[184981]: E1201 10:02:28.918400  184981 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ecs1503728529?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="6.4s"
Dec 01 10:02:30 ecs1503728529 k3s[184981]: E1201 10:02:30.812408  184981 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ecs1503728529\" not found"
Dec 01 10:02:34 ecs1503728529 k3s[184981]: E1201 10:02:34.990373  184981 server.go:304] "Unable to authenticate the request due to an error" err="context canceled"
Dec 01 10:02:40 ecs1503728529 k3s[184981]: E1201 10:02:40.813117  184981 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ecs1503728529\" not found"
Dec 01 10:02:45 ecs1503728529 k3s[184981]: W1201 10:02:45.320100  184981 transport.go:356] Unable to cancel request for *otelhttp.Transport
Dec 01 10:02:45 ecs1503728529 k3s[184981]: E1201 10:02:45.320444  184981 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ecs1503728529?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Dec 01 10:02:49 ecs1503728529 k3s[184981]: E1201 10:02:49.975063  184981 server.go:304] "Unable to authenticate the request due to an error" err="context canceled"
Dec 01 10:02:50 ecs1503728529 k3s[184981]: E1201 10:02:50.814221  184981 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ecs1503728529\" not found"

kubectl get nodes:

E1201 10:05:59.673656  185579 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1201 10:05:59.674435  185579 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1201 10:05:59.676407  185579 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1201 10:05:59.677055  185579 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1201 10:05:59.679021  185579 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused

[namehehe]

server logs:

Dec 01 10:12:55 ecs4342576676 k3s[270791]: time="2024-12-01T10:12:55Z" level=info msg="COMPACT deleted 683 rows from 688 revisions in 63.76287ms - compacted to 171343/172343"
Dec 01 10:13:09 ecs4342576676 k3s[270791]: E1201 10:13:09.756287  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0x7, Code:0x0, Cause:error(nil)}: stream error: stream ID 7; NO_ERROR
Dec 01 10:13:09 ecs4342576676 k3s[270791]: E1201 10:13:09.756638  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:13:09 ecs4342576676 k3s[270791]: E1201 10:13:09.758160  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:13:09 ecs4342576676 k3s[270791]: E1201 10:13:09.758230  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:13:09 ecs4342576676 k3s[270791]: E1201 10:13:09.759550  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.477905ms, POST "/api/v1/nodes" result: <nil>
Dec 01 10:13:55 ecs4342576676 k3s[270791]: E1201 10:13:55.037087  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0x7, Code:0x0, Cause:error(nil)}: stream error: stream ID 7; NO_ERROR
Dec 01 10:13:55 ecs4342576676 k3s[270791]: E1201 10:13:55.037349  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:13:55 ecs4342576676 k3s[270791]: E1201 10:13:55.038978  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:13:55 ecs4342576676 k3s[270791]: E1201 10:13:55.039034  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:13:55 ecs4342576676 k3s[270791]: E1201 10:13:55.040332  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.452103ms, POST "/api/v1/nodes" result: <nil>
Dec 01 10:16:10 ecs4342576676 k3s[270791]: E1201 10:16:10.699497  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0xd, Code:0x0, Cause:error(nil)}: stream error: stream ID 13; NO_ERROR
Dec 01 10:16:10 ecs4342576676 k3s[270791]: E1201 10:16:10.699794  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:16:10 ecs4342576676 k3s[270791]: E1201 10:16:10.701302  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:16:10 ecs4342576676 k3s[270791]: E1201 10:16:10.701357  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:16:10 ecs4342576676 k3s[270791]: E1201 10:16:10.702591  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.418453ms, POST "/api/v1/nodes" result: <nil>
Dec 01 10:17:44 ecs4342576676 k3s[270791]: E1201 10:17:44.392352  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0xd, Code:0x0, Cause:error(nil)}: stream error: stream ID 13; NO_ERROR
Dec 01 10:17:44 ecs4342576676 k3s[270791]: E1201 10:17:44.392691  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:17:44 ecs4342576676 k3s[270791]: E1201 10:17:44.394096  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:17:44 ecs4342576676 k3s[270791]: E1201 10:17:44.394151  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:17:44 ecs4342576676 k3s[270791]: E1201 10:17:44.395732  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.638195ms, POST "/api/v1/nodes" result: <nil>
Dec 01 10:17:55 ecs4342576676 k3s[270791]: time="2024-12-01T10:17:55Z" level=info msg="COMPACT compactRev=171343 targetCompactRev=172028 currentRev=173028"
Dec 01 10:17:55 ecs4342576676 k3s[270791]: time="2024-12-01T10:17:55Z" level=info msg="COMPACT deleted 683 rows from 685 revisions in 61.000879ms - compacted to 172028/173028"
Dec 01 10:18:29 ecs4342576676 k3s[270791]: E1201 10:18:29.422635  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0x9, Code:0x0, Cause:error(nil)}: stream error: stream ID 9; NO_ERROR
Dec 01 10:18:29 ecs4342576676 k3s[270791]: E1201 10:18:29.422946  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:18:29 ecs4342576676 k3s[270791]: E1201 10:18:29.424160  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:18:29 ecs4342576676 k3s[270791]: E1201 10:18:29.424225  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:18:29 ecs4342576676 k3s[270791]: E1201 10:18:29.425571  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.323238ms, POST "/api/v1/nodes" result: <nil>
Dec 01 10:19:14 ecs4342576676 k3s[270791]: E1201 10:19:14.491523  270791 status.go:71] apiserver received an error that is not an metav1.Status: http.http2StreamError{StreamID:0xb, Code:0x0, Cause:error(nil)}: stream error: stream ID 11; NO_ERROR
Dec 01 10:19:14 ecs4342576676 k3s[270791]: E1201 10:19:14.491805  270791 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Dec 01 10:19:14 ecs4342576676 k3s[270791]: E1201 10:19:14.493153  270791 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Dec 01 10:19:14 ecs4342576676 k3s[270791]: E1201 10:19:14.493208  270791 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Dec 01 10:19:14 ecs4342576676 k3s[270791]: E1201 10:19:14.494527  270791 timeout.go:142] post-timeout activity - time-elapsed: 3.402981ms, POST "/api/v1/nodes" result: <nil>
ec 01 10:20:16 ecs4342576676 k3s[270791]: I1201 10:20:16.324158  270791 node_controller.go:425] Initializing node ecs1503728529 with cloud provider
Dec 01 10:20:16 ecs4342576676 k3s[270791]: I1201 10:20:16.324435  270791 node_controller.go:229] error syncing 'ecs1503728529': failed to get instance metadata for node ecs1503728529: address annotations not yet set, requeuing
Dec 01 10:20:16 ecs4342576676 k3s[270791]: E1201 10:20:16.324472  270791 node_controller.go:240] error syncing 'ecs1503728529': failed to get instance metadata for node ecs1503728529: address annotations not yet set, requeuing
Dec 01 10:21:53 ec

server command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - 

[brandond]

I don't know what the network is like between your two nodes but it sounds like there are problems. Something is causing communication between nodes to drop and time out. Make sure you've opened all the correct ports between the nodes, and that you don't have any security agents on or between these nodes that might be interfering with connections.

It could also be that one or more of your nodes is overloaded, but I'm not seeing any of the other errors that I'd expect to see if you had insufficient CPU or memory.