Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]1.0 pulsar cluster created failed #8467

Closed
tianyue86 opened this issue Nov 15, 2024 · 2 comments · Fixed by apecloud/kubeblocks-addons#1255
Closed

[BUG]1.0 pulsar cluster created failed #8467

tianyue86 opened this issue Nov 15, 2024 · 2 comments · Fixed by apecloud/kubeblocks-addons#1255
Assignees
@shanshanying
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 1.0.0

Comments

@tianyue86
Copy link

tianyue86 commented Nov 15, 2024 •
edited
Loading

Describe the env

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.6
kbcli: 1.0.0-beta.3

To Reproduce
Steps to reproduce the behavior:

  1. Get the latest pulsar cluster yaml
helm template pulsarc2 ./addons-cluster/pulsar --version 1.0.0-alpha.0
---
# Source: pulsar-cluster/templates/cluster.yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: pulsarc2
  namespace: default
  labels: 
    helm.sh/chart: pulsar-cluster-1.0.0-alpha.0
    app.kubernetes.io/version: "3.0.2"
    app.kubernetes.io/instance: pulsarc2
  annotations:
    resource.kubeblocks.io/ignore-constraint: "true"
    ## Todo: use cluster api to control the rendering logic of service in component definition
    kubeblocks.io/enabled-pod-ordinal-svc: broker
    "kubeblocks.io/extra-env": '{"KB_PULSAR_BROKER_NODEPORT": "false"}'
spec:
  terminationPolicy: Delete
  services:
    - name: broker-bootstrap
      serviceName: broker-bootstrap
      componentSelector: broker
      spec:
        type: ClusterIP
        ports:
          - name: pulsar
            port: 6650
            targetPort: 6650
          - name: http
            port: 80
            targetPort: 8080
          - name: kafka-client
            port: 9092
            targetPort: 9092
    - name: zookeeper
      serviceName: zookeeper
      componentSelector: zookeeper
      spec:
        type: ClusterIP
        ports:
          - name: client
            port: 2181
            targetPort: 2181
  componentSpecs:
    - name: proxy
      componentDef: pulsar-proxy
      
      
      replicas: 3
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: bookies-recovery
      componentDef: pulsar-bookies-recovery
      
      
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: broker
      componentDef: pulsar-broker
      
      
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: bookies
      componentDef: pulsar-bookkeeper
      
      
      replicas: 4
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: zookeeper
      componentDef: pulsar-zookeeper
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "100m"
          memory: "512Mi"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Apply the yaml to create cluster
  2. Check the cluster status: Creating
NAMESPACE   NAME       CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
default     pulsarc2                                                        Delete                            Abnormal   30m
  1. check pod: some of pods are in Pending status
k get pod:
pulsarc2-bookies-1             2/2     Running                 0                3h11m
pulsarc2-bookies-2             2/2     Running                 0                3h11m
pulsarc2-bookies-3             2/2     Running                 0                3h11m
pulsarc2-bookies-recovery-0    1/1     Running                 0                3h11m
pulsarc2-broker-0              0/2     Init:CrashLoopBackOff   42 (48s ago)     3h10m
pulsarc2-proxy-0               1/1     Running                 0                3h10m
pulsarc2-proxy-1               1/1     Running                 0                3h10m
pulsarc2-proxy-2               1/1     Running                 0                3h10m
pulsarc2-zookeeper-0           1/1     Running                 0                3h10m

k describe pod pulsarc2-broker-0
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  BackOff  12m (x830 over 3h11m)  kubelet  Back-off restarting failed container init-pulsar-tools in pod pulsarc2-broker-0_default(abeb536b-44e9-4ebc-ad25-d30d8a43823b)
  Normal   Pulled   2m8s (x42 over 3h11m)  kubelet  Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11" already present on machine

--- check logs of container init-pulsar-tools
kubectl logs pulsarc2-broker-0 -c init-pulsar-tools --namespace default
cp: cannot stat '/bin/pulsar-admin': No such file or directory
  1. check pvc
k get pvc | grep pulsar
data-pulsarc2-zookeeper-0           Bound    d-8vbakckmgjaboa4mzwtv   20Gi       RWO            kb-default-sc   <unset>                 3h25m
journal-pulsarc2-bookies-0          Bound    d-8vbc4osgystir5l67m7d   20Gi       RWO            kb-default-sc   <unset>                 3h25m
journal-pulsarc2-bookies-1          Bound    d-8vbhiog6ec9nm6hjp3ge   20Gi       RWO            kb-default-sc   <unset>                 3h25m
journal-pulsarc2-bookies-2          Bound    d-8vbc1rfs0euzp420r8e1   20Gi       RWO            kb-default-sc   <unset>                 3h25m
journal-pulsarc2-bookies-3          Bound    d-8vbew4v9gfffkm1lee4e   20Gi       RWO            kb-default-sc   <unset>                 3h25m
ledgers-pulsarc2-bookies-0          Bound    d-8vb88aki0yj26z9yznjm   20Gi       RWO            kb-default-sc   <unset>                 3h25m
ledgers-pulsarc2-bookies-1          Bound    d-8vb89sdr8npg6mkvv6w7   20Gi       RWO            kb-default-sc   <unset>                 3h25m
ledgers-pulsarc2-bookies-2          Bound    d-8vb26m3ctr8qsmrac1nu   20Gi       RWO            kb-default-sc   <unset>                 3h25m
ledgers-pulsarc2-bookies-3          Bound    d-8vb9c7agv97fzda9oegb   20Gi       RWO            kb-default-sc   <unset>                 3h25m
  1. Check cmp
k get cmp|grep pulsar
pulsarc2-bookies             pulsar-bookkeeper-3-1.0.0-alpha.0         3.0.2             Running   3h25m
pulsarc2-bookies-recovery    pulsar-bookies-recovery-3-1.0.0-alpha.0   3.0.2             Running   3h25m
pulsarc2-broker              pulsar-broker-3-1.0.0-alpha.0             3.0.2             Failed    3h25m
pulsarc2-proxy               pulsar-proxy-3-1.0.0-alpha.0              3.0.2             Running   3h25m
pulsarc2-zookeeper           pulsar-zookeeper-3-1.0.0-alpha.0          3.0.2             Running   3h25m

k describe cmp pulsarc2-broker
Status:
  Conditions:
    Last Transition Time:  2024-11-26T04:21:18Z
    Message:               The operator has started the provisioning of Cluster: pulsarc2-broker
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2024-11-26T04:21:24Z
    Message:               the component phase is Failed
    Observed Generation:   1
    Reason:                Unavailable
    Status:                False
    Type:                  Available
  Message:
    InstanceSet/pulsarc2-broker:  ["pulsarc2-broker-0"]
  Observed Generation:            1
  Phase:                          Failed
Events:                           <none>
@tianyue86 tianyue86 added the kind/bug Something isn't working label Nov 15, 2024
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Nov 21, 2024
@tianyue86 tianyue86 added the severity/major Great chance user will encounter the same problem label Nov 21, 2024
@tianyue86 tianyue86 changed the title [BUG]1.0 Back-off restarting failed error occurred when creating pulsar cluster with the latest yaml [BUG]1.0 pulsar cluster created failed without status Nov 26, 2024
@tianyue86 tianyue86 changed the title [BUG]1.0 pulsar cluster created failed without status [BUG]1.0 pulsar cluster created failed Nov 26, 2024
@tianyue86
Copy link
Author

@caiq1nyu please help investigate. Thanks!

@shanshanying
Copy link
Contributor

i wiil do the fix.

@shanshanying shanshanying mentioned this issue Nov 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shanshanying shanshanying

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 1.0.0
Development

Successfully merging a pull request may close this issue.

fix: update pulsar cmpv and cc apecloud/kubeblocks-addons
4 participants
@caiq1nyu @Y-Rookie @shanshanying @tianyue86