Issues in Redis to valkey upgrade

[royatanu94]

Hi All,
Getting issue while upgrading redis to valkey in K8s. But it works fine when we fresh install it.

Getting error:
cess: exec: \"valkey-server\": executable file not found in $PATH" Warning Failed 19m kubelet Error: container create failed: time="2024-10-23T07:10:33Z" level=error msg="runc create failed: unable to start container process: exec: \"valkey-server\": executable file not found in $PATH" Warning Failed 19m kubelet Error: container create failed: time="2024-10-23T07:10:47Z" level=error msg="runc create failed: unable to start container process: exec: \"valkey-server\": executable file not found in $PATH"
Please help me to fix it.

[royatanu94]

@rueian @zuiderkwast
Any help is appreciated.

[royatanu94]

@rueian Any suggestion to fix it ? Are you talking about "kube-dns-*" pod which might be running on the worker node which I rebooted ? I also have doubts like core-DNS or Kube-DNS is doing host dns resolution

[rueian]

Are you talking about "kube-dns-*" pod which might be running on the worker node which I rebooted ?

Yes, that is likely the cause. NodeLocal DNSCache https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/ might be helpful.

[royatanu94]

@rueian the worker node running valley did not have any DNS pod restarts so not getting why its showing sentinel address unable to resolve.

[royatanu94]

@rueian From logs its seen that my application replica1 discoveres below sentinels
redis: 2024/10/25 06:48:18 sentinel.go:742: sentinel: discovered new sentinel="sentinel-2.sentinel:26379" for master="master" redis: 2024/10/25 06:48:18 sentinel.go:742: sentinel: discovered new sentinel="sentinel-1.sentinel:26379" for master="master"
&
replica2 discoveres
redis: 2024/10/25 06:47:55 sentinel.go:742: sentinel: discovered new sentinel="sentinel-2.sentinel:26379" for master="master" redis: 2024/10/25 06:47:55 sentinel.go:742: sentinel: discovered new sentinel="sentinel-0.sentinel:26379" for master="master"
Both replica1 and replica2 running in different worker node. Valkey is runnning 3master and 3 sentinel instances in different node.

So, If i reboot my application replica2 worker node, I saw replica1 unable to reach sentinel-2 & sentinel-1, once after timeouts then it discoveres new sentinel-0 and with that connection works fine.

Immediately after reboot of application replica2 node I have seen below sentinel-0 logs
``
1:X 25 Oct 2024 06:59:25.167 # Failed to resolve hostname 'sentinel-1.sentinel'
1:X 25 Oct 2024 06:59:25.168 # +tilt #tilt mode entered
1:X 25 Oct 2024 06:59:35.207 # Failed to resolve hostname 'sentinel-2.sentinel'
1:X 25 Oct 2024 06:59:35.207 # +tilt #tilt mode entered
1:X 25 Oct 2024 06:59:45.255 # Failed to resolve hostname 'sentinel-2.sentinel'

Go-redis log in application replica1 :
redis: 2024/10/25 06:59:34 sentinel.go:558: sentinel: GetMasterAddrByName master="master" failed: dial tcp: lookup sentinel-2.sentinel: i/o timeout

[rueian]

Can you try deploying your application and valley sentinel to a well-configured K8s? Such as AWS EKS, Azure AKE, or GCP GKE.