Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]1.0 elasticsearch cluster created failed: Readiness probe failed #8481

Closed
tianyue86 opened this issue Nov 19, 2024 · 3 comments · Fixed by apecloud/kubeblocks-addons#1251
Closed

[BUG]1.0 elasticsearch cluster created failed: Readiness probe failed #8481

tianyue86 opened this issue Nov 19, 2024 · 3 comments · Fixed by apecloud/kubeblocks-addons#1251
Assignees
@iziang @Y-Rookie @shanshanying
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 1.0.0

Comments

@tianyue86
Copy link

tianyue86 commented Nov 19, 2024 •
edited
Loading

Describe the env

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.6
kbcli: 1.0.0-beta.3

To Reproduce
Steps to reproduce the behavior:

  1. Get the latest es cluster yaml
helm template esclu02 ./addons-cluster/elasticsearch --version 1.0.0-alpha.0
---
# Source: elasticsearch-cluster/templates/cluster-multi-node.yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: elastic2
  namespace: default
  labels:
    helm.sh/chart: elasticsearch-cluster-1.0.0-alpha.0
    app.kubernetes.io/version: "8.8.2"
    app.kubernetes.io/instance: elastic2
  annotations:
    kubeblocks.io/extra-env: '{"mdit-roles":"master,data,ingest,transform","mode":"multi-node"}'
spec:
  terminationPolicy: Delete
  componentSpecs:
    - name: mdit
      componentDef: elasticsearch-8
      serviceVersion: 8.8.2
      serviceAccountName: kb-elastic2     
      schedulingPolicy:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: elastic2
                    apps.kubeblocks.io/component-name: mdit
                topologyKey: kubernetes.io/hostname
              weight: 100
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: elastic2
                  apps.kubeblocks.io/component-name: mdit
              topologyKey: kubernetes.io/hostname     
      replicas: 3     
      disableExporter: false     
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
        requests:
          cpu: "1"
          memory: "2Gi"     
      volumeClaimTemplates:
        - name: data # ref clusterDefinition components.containers.volumeMounts.name
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Check the cluster status: Failed
k get cluster -A
NAMESPACE   NAME       CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS   AGE
default     elastic2                        Delete               Failed   40m
  1. Check Pod status: CrashLoopBackOff
k get pod
NAME              READY   STATUS             RESTARTS        AGE
elastic2-mdit-0   2/3     CrashLoopBackOff   10 (3m5s ago)   40m
elastic2-mdit-1   2/3     CrashLoopBackOff   11 (19s ago)    40m
elastic2-mdit-2   2/3     CrashLoopBackOff   11 (21s ago)    40m
  1. Describe pod
Events:
  Type     Reason                  Age    From                     Message
  ----     ------                  ----   ----                     -------
  Normal   Scheduled               9m47s  default-scheduler        Successfully assigned default/elastic2-mdit-2 to cn-zhangjiakou.10.0.0.140
  Normal   SuccessfulAttachVolume  9m47s  attachdetach-controller  AttachVolume.Attach succeeded for volume "d-8vb72lgx871r2aagx87u"
  Normal   AllocIPSucceed          9m38s  terway-daemon            Alloc IP 10.0.0.170/24 took 32.052877ms
  Normal   Pulled                  9m38s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch-plugins:8.8.2" already present on machine
  Normal   Created                 9m38s  kubelet                  Created container prepare-plugins
  Normal   Started                 9m38s  kubelet                  Started container prepare-plugins
  Normal   Pulled                  9m37s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch:8.8.2" already present on machine
  Normal   Created                 9m37s  kubelet                  Created container install-plugins
  Normal   Started                 9m37s  kubelet                  Started container install-plugins
  Normal   Pulled                  9m36s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.0-beta.6" already present on machine
  Normal   Created                 9m36s  kubelet                  Created container init-kbagent
  Normal   Started                 9m36s  kubelet                  Started container init-kbagent
  Normal   Pulled                  9m35s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/curl-jq:0.1.0" already present on machine
  Normal   Created                 9m35s  kubelet                  Created container kbagent-worker
  Normal   Started                 9m35s  kubelet                  Started container kbagent-worker
  Normal   Pulled                  9m34s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch:8.8.2" already present on machine
  Normal   Created                 9m34s  kubelet                  Created container elasticsearch
  Normal   Started                 9m34s  kubelet                  Started container elasticsearch
  Normal   Pulled                  9m34s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch-exporter:v1.7.0" already present on machine
  Normal   Created                 9m34s  kubelet                  Created container exporter
  Normal   Started                 9m34s  kubelet                  Started container exporter
  Normal   Pulled                  9m34s  kubelet                  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/curl-jq:0.1.0" already present on machine
  Normal   Created                 9m34s  kubelet                  Created container kbagent
  Normal   Started                 9m34s  kubelet                  Started container kbagent
  Warning  Unhealthy               9m29s  kubelet                  Liveness probe failed: Get "http://10.0.0.170:9114/healthz": dial tcp 10.0.0.170:9114: connect: connection refused
  Warning  Unhealthy               9m29s  kubelet                  Readiness probe failed: Get "http://10.0.0.170:9114/healthz": dial tcp 10.0.0.170:9114: connect: connection refused
  Warning  Unhealthy               9m23s  kubelet                  Readiness probe failed: {"timestamp": "2024-11-26 02:11:39", "message": "readiness probe failed", "curl_rc": "7"}
readiness probe check failed
  Warning  Unhealthy  9m18s  kubelet  Readiness probe failed: {"timestamp": "2024-11-26 02:11:44", "message": "readiness probe failed", "curl_rc": "7"}
readiness probe check failed
  Warning  Unhealthy  4m34s (x26 over 8m49s)  kubelet  (combined from similar events): Readiness probe failed: {"timestamp": "2024-11-26 02:16:28", "message": "readiness probe failed", "curl_rc": "7"}
readiness probe check failed
  1. get pvc
data-elastic2-mdit-0                Bound    d-8vb5zxu58ahyy569ssvd   20Gi       RWO            kb-default-sc   <unset>                 25m
data-elastic2-mdit-1                Bound    d-8vb185btd96viw0iwu9b   20Gi       RWO            kb-default-sc   <unset>                 25m
data-elastic2-mdit-2                Bound    d-8vb72lgx871r2aagx87u   20Gi       RWO            kb-default-sc   <unset>                 25m
  1. check cmp
elastic2-mdit                elasticsearch-8-1.0.0-alpha.0             8.8.2             Failed    26m

k describe cmp:
Events:
  Type    Reason                    Age                From                  Message
  ----    ------                    ----               ----                  -------
  Normal  Unknown                   26m                component-controller  the component phase is unknown
  Normal  ComponentPhaseTransition  26m (x3 over 26m)  component-controller  component is Creating
  Normal  Unavailable               26m (x3 over 26m)  component-controller  the component phase is Creating
  Normal  ComponentPhaseTransition  24m (x3 over 26m)  component-controller  component is Updating
  Normal  Unavailable               24m (x3 over 26m)  component-controller  the component phase is Updating
  Normal  ComponentPhaseTransition  24m (x4 over 26m)  component-controller  component is Failed
  Normal  Unavailable               24m (x4 over 26m)  component-controller  the component phase is Failed

Attach the logs for investigation:

kblogs.zip
@tianyue86 tianyue86 added the kind/bug Something isn't working label Nov 19, 2024
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Nov 19, 2024
@tianyue86 tianyue86 added the severity/major Great chance user will encounter the same problem label Nov 22, 2024
@tianyue86 tianyue86 changed the title [BUG]1.0 elasticsearch cluster created failed: Back-off restarting failed container [BUG]1.0 elasticsearch cluster created failed: Readiness probe failed Nov 26, 2024
@tianyue86
Copy link
Author

Updated the issue description based on the latest kb version.

@iziang
Copy link
Contributor

iziang commented Nov 27, 2024

The KB operator keeps crashing, please address this issue first.

2024-11-27T01:45:18.100Z	ERROR	ObjectTreeRootFinder	list trace failed	{"error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:19.102Z	DPANIC	ObjectTreeRootFinder	odd number of arguments passed as key-value pairs for logging	{"ignored key": ""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:19.102Z	ERROR	ObjectTreeRootFinder	list trace failed	{"error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:20.101Z	DPANIC	ObjectTreeRootFinder	odd number of arguments passed as key-value pairs for logging	{"ignored key": ""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:20.101Z	ERROR	ObjectTreeRootFinder	list trace failed	{"error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:21.102Z	DPANIC	ObjectTreeRootFinder	odd number of arguments passed as key-value pairs for logging	{"ignored key": ""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:21.102Z	ERROR	ObjectTreeRootFinder	list trace failed	{"error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:22.105Z	DPANIC	ObjectTreeRootFinder	odd number of arguments passed as key-value pairs for logging	{"ignored key": ""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:22.105Z	ERROR	ObjectTreeRootFinder	list trace failed	{"error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
github.com/apecloud/kubeblocks/controllers/trace.(*rootFinder).findRoots
	/src/controllers/trace/object_tree_root_finder.go:134
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).mapAndEnqueue
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:81
sigs.k8s.io/controller-runtime/pkg/handler.(*enqueueRequestsFromMapFunc).Generic
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/handler/enqueue_mapped.go:77
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:133
sigs.k8s.io/controller-runtime/pkg/source.(*Channel).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:134
2024-11-27T01:45:23.034Z	ERROR	Could not wait for Cache to sync	{"controller": "reconciliationtrace", "controllerGroup": "trace.kubeblocks.io", "controllerKind": "ReconciliationTrace", "error": "failed to wait for reconciliationtrace caches to sync: timed out waiting for cache to be synced for Kind *v1.ReconciliationTrace"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:203
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:223
2024-11-27T01:45:23.034Z	INFO	Stopping and waiting for non leader election runnables
2024-11-27T01:45:23.034Z	INFO	Stopping and waiting for leader election runnables
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "componentversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentVersion"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "shardingdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ShardingDefinition"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "sidecardefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "SidecarDefinition"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "opsrequest", "controllerGroup": "operations.kubeblocks.io", "controllerKind": "OpsRequest"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "opsdefinition", "controllerGroup": "operations.kubeblocks.io", "controllerKind": "OpsDefinition"}
2024-11-27T01:45:23.034Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet"}
2024-11-27T01:45:23.035Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "opsdefinition", "controllerGroup": "operations.kubeblocks.io", "controllerKind": "OpsDefinition"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "sidecardefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "SidecarDefinition"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "opsrequest", "controllerGroup": "operations.kubeblocks.io", "controllerKind": "OpsRequest"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "componentversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentVersion"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "shardingdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ShardingDefinition"}
2024-11-27T01:45:23.035Z	INFO	All workers finished	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2024-11-27T01:45:23.035Z	INFO	Stopping and waiting for caches
W1127 01:45:23.035199       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.BackupPolicyTemplate ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035243       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.RoleBinding ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035252       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.Lease ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035280       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.SidecarDefinition ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035296       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1beta1.ConfigConstraint ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035329       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.ClusterDefinition ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035340       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1beta1.VolumeSnapshot ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035354       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.ServiceDescriptor ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035366       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.PersistentVolume ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035199       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.Event ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035394       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.VolumeSnapshot ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035442       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.Addon ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035466       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.Backup ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035492       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.OpsRequest ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035517       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.OpsDefinition ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035544       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035560       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1alpha1.Restore ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W1127 01:45:23.035602       1 reflector.go:462] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: watch of *v1.Job ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-11-27T01:45:23.036Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "ReconciliationTrace.trace.kubeblocks.io", "error": "no matches for kind \"ReconciliationTrace\" in version \"trace.kubeblocks.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:63
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56
2024-11-27T01:45:23.036Z	INFO	Stopping and waiting for webhooks
2024-11-27T01:45:23.036Z	INFO	Stopping and waiting for HTTP servers
2024-11-27T01:45:23.036Z	INFO	shutting down server	{"kind": "health probe", "addr": "[::]:8081"}
2024-11-27T01:45:23.036Z	INFO	controller-runtime.metrics	Shutting down metrics server with timeout of 1 minute
2024-11-27T01:45:23.036Z	INFO	Wait completed, proceeding to shutdown the manager
2024-11-27T01:45:23.042Z	ERROR	error received after stop sequence was engaged	{"error": "leader election lost"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:490
2024-11-27T01:45:23.042Z	ERROR	setup	problem running manager	{"error": "failed to wait for reconciliationtrace caches to sync: timed out waiting for cache to be synced for Kind *v1.ReconciliationTrace"}
main.main
	/src/cmd/manager/main.go:625
runtime.main
	/usr/local/go/src/runtime/proc.go:267

@iziang
Copy link
Contributor

iziang commented Nov 27, 2024 •
edited
Loading

POD_FQDN env是错的，应该是elastic2-mdit-0.elastic2-mdit-headless.default.svc.cluster.local
image

image

@iziang iziang linked a pull request Nov 27, 2024 that will close this issue
fix[elasticsearch]: invalid pod fqdn apecloud/kubeblocks-addons#1251
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iziang iziang

@Y-Rookie Y-Rookie

@shanshanying shanshanying

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 1.0.0
Development

Successfully merging a pull request may close this issue.

fix[elasticsearch]: invalid pod fqdn apecloud/kubeblocks-addons
4 participants
@iziang @Y-Rookie @shanshanying @tianyue86