Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] edis cluster pod crash kill 1 in pod #8511

Open
JashBook opened this issue Nov 22, 2024 · 0 comments
Open

[BUG] edis cluster pod crash kill 1 in pod #8511

JashBook opened this issue Nov 22, 2024 · 0 comments
Assignees
@Y-Rookie
Labels
kind/bug Something isn't working

Comments

@JashBook
Copy link
Collaborator

Describe the bug

kbcli version
Kubernetes: v1.30.6
KubeBlocks: 0.9.2-beta.22
kbcli: 0.9.1

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: redis-ttznhh
  namespace: default
spec:
  terminationPolicy: Delete
  componentSpecs:
    - name: redis
      componentDef: redis-7
      replicas: 2
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      switchPolicy:
        type: Noop
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: redis-sentinel
      componentDef: redis-sentinel-7
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    
    - name: redis-twemproxy
      componentDef: redis-twemproxy-0.5
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
  1. kiil 1
kubectl exec -it redis-ttznhh-redis-0 bash
kiil 1
  1. See error
kubectl get cluster
NAME             CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
redis-ttznhh                                    Delete               Abnormal   5h30m

kubectl get pod 
NAME                             READY   STATUS             RESTARTS       AGE
redis-ttznhh-redis-0             2/3     CrashLoopBackOff   63 (71s ago)   5h
redis-ttznhh-redis-1             3/3     Running            0              5h
redis-ttznhh-redis-sentinel-0    2/2     Running            0              5h18m
redis-ttznhh-redis-sentinel-1    2/2     Running            0              5h30m
redis-ttznhh-redis-sentinel-2    2/2     Running            0              4h6m
redis-ttznhh-redis-twemproxy-0   1/1     Running            0              5h8m
redis-ttznhh-redis-twemproxy-1   1/1     Running            0              4h6m

describe pod

kubectl describe pod redis-ttznhh-redis-0
Name:             redis-ttznhh-redis-0
Namespace:        default
Priority:         0
Service Account:  kb-redis-ttznhh
Node:             aks-cicdamdpool-23244420-vmss000005/10.224.0.6
Start Time:       Fri, 22 Nov 2024 11:14:22 +0800
Labels:           app.kubernetes.io/component=redis-7
                  app.kubernetes.io/instance=redis-ttznhh
                  app.kubernetes.io/managed-by=kubeblocks
                  app.kubernetes.io/name=redis-7
                  app.kubernetes.io/version=redis-7
                  apps.kubeblocks.io/cluster-uid=894c9367-4ece-4540-912e-41b3b5e1ab17
                  apps.kubeblocks.io/component-name=redis
                  apps.kubeblocks.io/pod-name=redis-ttznhh-redis-0
                  componentdefinition.kubeblocks.io/name=redis-7
                  controller-revision-hash=5bdcd4697b
                  kubeblocks.io/role=primary
                  workloads.kubeblocks.io/access-mode=ReadWrite
                  workloads.kubeblocks.io/instance=redis-ttznhh-redis
                  workloads.kubeblocks.io/managed-by=InstanceSet
Annotations:      apps.kubeblocks.io/component-replicas: 2
                  apps.kubeblocks.io/last-role-snapshot-version: 1732245306492198
                  config.kubeblocks.io/restart-redis-replication-config: 5b7b596b7f
Status:           Running
IP:               10.244.1.23
IPs:
  IP:           10.244.1.23
Controlled By:  InstanceSet/redis-ttznhh-redis
Init Containers:
  init-lorry:
    Container ID:  containerd://e73d5be318b68e292688d84131ce90ece3815cd703c567a183029f57ab526c31
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.2-beta.22
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:9f6e076fac2b480a49f4610a55d92dea3e11c74623f1d097da5d86a26264ac39
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      -r
      /bin/lorry
      /config
      /bin/curl
      /kubeblocks/
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 22 Nov 2024 11:14:48 +0800
      Finished:     Fri, 22 Nov 2024 11:14:48 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      redis-ttznhh-redis-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:      <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:  <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      SENTINEL_USER:           <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:       <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:             redis-ttznhh-redis-0 (v1:metadata.name)
      KB_POD_UID:               (v1:metadata.uid)
      KB_NAMESPACE:            default (v1:metadata.namespace)
      KB_SA_NAME:               (v1:spec.serviceAccountName)
      KB_NODENAME:              (v1:spec.nodeName)
      KB_HOST_IP:               (v1:status.hostIP)
      KB_POD_IP:                (v1:status.podIP)
      KB_POD_IPS:               (v1:status.podIPs)
      KB_HOSTIP:                (v1:status.hostIP)
      KB_PODIP:                 (v1:status.podIP)
      KB_PODIPS:                (v1:status.podIPs)
      KB_POD_FQDN:             $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
    Mounts:
      /kubeblocks from kubeblocks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
Containers:
  redis:
    Container ID:  containerd://3c8d5db460ec84d9faa3eb7860cc48af2de0a050f05b6de00f61dd64a7faab00
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
    Port:          6379/TCP
    Host Port:     0/TCP
    Command:
      /scripts/redis-start.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 22 Nov 2024 16:14:06 +0800
      Finished:     Fri, 22 Nov 2024 16:14:06 +0800
    Ready:          False
    Restart Count:  63
    Limits:
      cpu:     100m
      memory:  512Mi
    Requests:
      cpu:      100m
      memory:   512Mi
    Readiness:  exec [sh -c /scripts/redis-ping.sh 1] delay=10s timeout=1s period=5s #success=1 #failure=5
    Environment Variables from:
      redis-ttznhh-redis-env      ConfigMap  Optional: false
      redis-ttznhh-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:       <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:   <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      SENTINEL_USER:            <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:        <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:              redis-ttznhh-redis-0 (v1:metadata.name)
      KB_POD_UID:                (v1:metadata.uid)
      KB_NAMESPACE:             default (v1:metadata.namespace)
      KB_SA_NAME:                (v1:spec.serviceAccountName)
      KB_NODENAME:               (v1:spec.nodeName)
      KB_HOST_IP:                (v1:status.hostIP)
      KB_POD_IP:                 (v1:status.podIP)
      KB_POD_IPS:                (v1:status.podIPs)
      KB_HOSTIP:                 (v1:status.hostIP)
      KB_PODIP:                  (v1:status.podIP)
      KB_PODIPS:                 (v1:status.podIPs)
      KB_POD_FQDN:              $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:             6379
      REDIS_REPL_USER:          kbreplicator
      REDIS_REPL_PASSWORD:      $(REDIS_DEFAULT_PASSWORD)
      REDIS_SENTINEL_USER:      $(REDIS_REPL_USER)-sentinel
      REDIS_SENTINEL_PASSWORD:  $(REDIS_DEFAULT_PASSWORD)
    Mounts:
      /data from data (rw)
      /etc/conf from redis-config (rw)
      /etc/redis from redis-conf (rw)
      /kb-podinfo from pod-info (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
  metrics:
    Container ID:  containerd://3735a9337c963f1dd105404f1b0a4759741226d01e5357203f303c31650ec93b
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:01d06a16a2e96d0c5e1ac97db40353abad8ee73f34f9f7d148984e7de4ff0bd5
    Port:          9121/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/conf/metrics-config.yaml
    State:          Running
      Started:      Fri, 22 Nov 2024 11:14:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      redis-ttznhh-redis-env      ConfigMap  Optional: false
      redis-ttznhh-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:      <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:  <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      SENTINEL_USER:           <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:       <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:             redis-ttznhh-redis-0 (v1:metadata.name)
      KB_POD_UID:               (v1:metadata.uid)
      KB_NAMESPACE:            default (v1:metadata.namespace)
      KB_SA_NAME:               (v1:spec.serviceAccountName)
      KB_NODENAME:              (v1:spec.nodeName)
      KB_HOST_IP:               (v1:status.hostIP)
      KB_POD_IP:                (v1:status.podIP)
      KB_POD_IPS:               (v1:status.podIPs)
      KB_HOSTIP:                (v1:status.hostIP)
      KB_PODIP:                 (v1:status.podIP)
      KB_PODIPS:                (v1:status.podIPs)
      KB_POD_FQDN:             $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
      ENDPOINT:                localhost:6379
      REDIS_USER:              $(REDIS_DEFAULT_USER)
      REDIS_PASSWORD:          $(REDIS_DEFAULT_PASSWORD)
    Mounts:
      /opt/conf from redis-metrics-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
  lorry:
    Container ID:  containerd://ac709cfbbb4edbc3392c8208274a95d1dbf29a926141e99aecdb95df129f6b63
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /kubeblocks/lorry
      --port
      3501
      --grpcport
      50001
      --config-path
      /kubeblocks/config/lorry/components/
    State:          Running
      Started:      Fri, 22 Nov 2024 11:14:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      redis-ttznhh-redis-env      ConfigMap  Optional: false
      redis-ttznhh-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:            <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:        <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'>           Optional: false
      SENTINEL_USER:                 <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:             <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:                   redis-ttznhh-redis-0 (v1:metadata.name)
      KB_POD_UID:                     (v1:metadata.uid)
      KB_NAMESPACE:                  default (v1:metadata.namespace)
      KB_SA_NAME:                     (v1:spec.serviceAccountName)
      KB_NODENAME:                    (v1:spec.nodeName)
      KB_HOST_IP:                     (v1:status.hostIP)
      KB_POD_IP:                      (v1:status.podIP)
      KB_POD_IPS:                     (v1:status.podIPs)
      KB_HOSTIP:                      (v1:status.hostIP)
      KB_PODIP:                       (v1:status.podIP)
      KB_PODIPS:                      (v1:status.podIPs)
      KB_POD_FQDN:                   $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
      KB_RSM_ROLE_PROBE_PERIOD:      1
      KB_BUILTIN_HANDLER:            redis
      KB_SERVICE_USER:               <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'>  Optional: false
      KB_SERVICE_PASSWORD:           <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'>  Optional: false
      KB_SERVICE_PORT:               6379
      KB_DATA_PATH:                  /data
      KB_ACTION_COMMANDS:            {"accountProvision":["sh","-c","redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD $(KB_ACCOUNT_STATEMENT) \u0026\u0026 redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD acl save "],"postProvision":["/bin/bash","-c","/scripts/redis-register-to-sentinel.sh"]}
      SERVICE_PORT:                  6379
      REDIS_REPL_USER:               kbreplicator
      REDIS_REPL_PASSWORD:           $(REDIS_DEFAULT_PASSWORD)
      REDIS_SENTINEL_USER:           $(REDIS_REPL_USER)-sentinel
      REDIS_SENTINEL_PASSWORD:       $(REDIS_DEFAULT_PASSWORD)
      KB_RSM_ACTION_SVC_LIST:        null
      KB_RSM_ROLE_UPDATE_MECHANISM:  DirectAPIServerEventUpdate
      KB_RSM_ROLE_PROBE_TIMEOUT:     1
      KB_CLUSTER_NAME:                (v1:metadata.labels['app.kubernetes.io/instance'])
      KB_COMP_NAME:                   (v1:metadata.labels['apps.kubeblocks.io/component-name'])
      KB_SERVICE_CHARACTER_TYPE:     redis
    Mounts:
      /data from data (rw)
      /etc/conf from redis-config (rw)
      /etc/redis from redis-conf (rw)
      /kb-podinfo from pod-info (rw)
      /kubeblocks from kubeblocks (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  redis-metrics-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-ttznhh-redis-redis-metrics-config
    Optional:  false
  redis-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-ttznhh-redis-redis-replication-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-ttznhh-redis-redis-scripts
    Optional:  false
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-redis-ttznhh-redis-0
    ReadOnly:   false
  redis-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kubeblocks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-dl5xk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From     Message
  ----     ------     ----                 ----     -------
  Normal   checkRole  60m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  55m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  50m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  45m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  40m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  35m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  30m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  25m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  20m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  15m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  10m                  lorry    {"operation":"checkRole","originalRole":"primary"}
  Normal   checkRole  5m20s                lorry    {"operation":"checkRole","originalRole":"primary"}
  Warning  BackOff    94s (x1423 over 5h)  kubelet  Back-off restarting failed container redis in pod redis-ttznhh-redis-0_default(4313e357-ce1f-495a-9b61-49486537a001)
  Normal   checkRole  20s                  lorry    {"operation":"checkRole","originalRole":"primary"}

logs pod

kubectl logs redis-ttznhh-redis-0 -c redis --previous
+ declare -g primary
+ declare -g primary_port
+ declare -g default_initialize_pod_ordinal
+ declare -g headless_postfix=headless
+ parse_redis_advertised_svc_if_exist redis-ttznhh-redis-0
+ local pod_name=redis-ttznhh-redis-0
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
+ build_redis_conf
+ load_redis_template_conf
+ echo 'include /etc/conf/redis.conf'
+ build_announce_ip_and_port
+ '[' -n '' ']'
+ kb_pod_fqdn=redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc
+ echo 'redis use kb pod fqdn redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc to announce'
+ echo 'replica-announce-ip redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc'
+ build_redis_service_port
+ service_port=6379
+ '[' -n 6379 ']'
+ service_port=6379
+ echo 'port 6379'
+ build_replicaof_config
+ init_or_get_primary_node
+ init_or_get_primary_from_redis_sentinel
+ '[' -z redis-ttznhh-redis-sentinel ']'
+ '[' -z redis-ttznhh-redis-sentinel-0,redis-ttznhh-redis-sentinel-1,redis-ttznhh-redis-sentinel-2 ']'
+ '[' -z redis-ttznhh-redis-sentinel-headless ']'
+ old_ifs=' 	
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS=' 	
'
+ declare -A master_count_map
+ local first_redis_primary_host=
+ local first_redis_primary_port=
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
redis use kb pod fqdn redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc to announce
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
sentinel:redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ echo 'sentinel:redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=1
+ [[ -z '' ]]
+ [[ -z '' ]]
+ first_redis_primary_host=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc
+ first_redis_primary_port=6379
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ echo 'sentinel:redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
sentinel:redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=2
+ [[ -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ]]
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
sentinel:redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ echo 'sentinel:redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=3
+ [[ -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ]]
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
get all primary info from redis sentinel master_count_map: 3
+ echo 'get all primary info from redis sentinel master_count_map: 3'
+ '[' 1 -eq 0 ']'
+ max_count=0
+ for host_port in "${!master_count_map[@]}"
+ ((  3 > max_count  ))
+ max_count=3
++ cut -d: -f1
++ echo redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ primary=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc
++ echo redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
++ cut -d: -f2
+ primary_port=6379
+ check_current_pod_is_primary
+ current_pod=redis-ttznhh-redis-0.redis-ttznhh-redis
+ [[ redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc == *\r\e\d\i\s\-\t\t\z\n\h\h\-\r\e\d\i\s\-\0\.\r\e\d\i\s\-\t\t\z\n\h\h\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc == \1\0\.\2\4\4\.\1\.\2\3 ]]
+ return 1
+ echo 'replicaof redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ rebuild_redis_acl_file
+ '[' -f /data/users.acl ']'
+ sed -i '/user default on/d' /data/users.acl
+ sed -i '/user kbreplicator on/d' /data/users.acl
+ sed -i '/user kbreplicator-sentinel on/d' /data/users.acl
+ build_redis_default_accounts
+ set +x
build default accounts succeeded!
+ echo 'aclfile /data/users.acl'
+ echo 'build default accounts succeeded!'
+ start_redis_server
+ exec_cmd='exec redis-server /etc/redis/redis.conf'
+ '[' -f /opt/redis-stack/lib/redisearch.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so '
+ '[' -f /opt/redis-stack/lib/redistimeseries.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so '
+ '[' -f /opt/redis-stack/lib/rejson.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so '
+ '[' -f /opt/redis-stack/lib/redisbloom.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so '
+ '[' -f /opt/redis-stack/lib/redisgraph.so ']'
+ '[' -f /opt/redis-stack/lib/rediscompat.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so'
+ '[' -f /opt/redis-stack/lib/redisgears.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so 
+ echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
+ eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Y-Rookie Y-Rookie

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Y-Rookie @JashBook