VolumeAffinityUpdated: Affinity was out of sync with LINSTOR resource . Data on the volume is lost!

[alexminder]

I1201 19:31:34.303493       1 controller.go:215] Need to replace PV 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8'
I1201 19:31:34.846833       1 controller.go:137] failed to reconcile resource: failed to replace PV: failed to update property value: Message: 'Successfully set property key(s): Aux/affinity-updater-saved-pv' next error: Message: 'Resource definition 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' modified.'; Details: 'Resource definition 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' UUID is: 89b79f20-d6a0-4fa9-98ce-0a2f93650323' next error: Message: '(okd-node-05) Resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' [DRBD] deleted.' next error: Message: '(okd-node-10) Resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' [DRBD] deleted.' next error: Message: '(Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.'; Cause: 'Verification of resource file failed'; Details: 'The error reported by the runtime environment or operating system is:
Resource definition: pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8'; Reports: '[6374ECDC-00000-020347,6374F2B0-62487-000017]'

$ oc describe pv pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Name:              pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
                   volume.kubernetes.io/provisioner-deletion-secret-name:
                   volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers:        [kubernetes.io/pv-protection external-attacher/linstor-csi-linbit-com]
StorageClass:      ssd
Status:            Bound
Claim:             NS/PVC
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          5Gi
Node Affinity:
  Required Terms:
    Term 0:        linbit.com/hostname in [okd-node-10]
    Term 1:        linbit.com/hostname in [okd-node-18]
    Term 2:        linbit.com/hostname in [okd-node-19]
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            linstor.csi.linbit.com
    FSType:            xfs
    VolumeHandle:      pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
    ReadOnly:          false
    VolumeAttributes:      linstor.csi.linbit.com/mount-options=noatime
                           linstor.csi.linbit.com/post-mount-xfs-opts=
                           linstor.csi.linbit.com/remote-access-policy=false
                           linstor.csi.linbit.com/uses-volume-context=true
                           storage.kubernetes.io/csiProvisionerIdentity=1668600378626-8081-linstor.csi.linbit.com
Events:                <none>

 $ oc describe pvc -n NS PVC
 ...
Status:        Bound
Volume:        pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               resize.topolvm.io/increase: 25%
               resize.topolvm.io/storage_limit: 100Gi
               resize.topolvm.io/threshold: 20%
               volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
               volume.kubernetes.io/selected-node: okd-node-18
               volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       POD
Events:
  Type     Reason                 Age                 From                         Message
  ----     ------                 ----                ----                         -------
  Normal   VolumeAffinityUpdated  130m                LinstorAffinityController    Affinity was out of sync with LINSTOR resource state
  Warning  ClaimLost              130m (x3 over 15h)  persistentvolume-controller  Bound claim has lost its PersistentVolume. Data on the volume is lost!

ERROR REPORT 6374ECDC-00000-020347

============================================================

Application:                        LINBIT�� LINSTOR
Module:                             Controller
Version:                            1.20.0
Build ID:                           9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time:                         2022-10-18T07:19:30+00:00
Error time:                         2022-12-01 19:31:34
Node:                               linstor-piraeus-cs-controller-6b47864974-rwts8
Peer:                               RestClient(ip; 'Go-http-client/1.1')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337

Error message:                      (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.

Error context:
  (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.

ApiRcException entries:
Nr: 1
Message: (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.
Details: The error reported by the runtime environment or operating system is:
The external command 'drbdadm' exited with error code 10

Cause: Verification of resource file failed

Asynchronous stage backtrace:

  Error has been observed at the following site(s):
  	|_ checkpoint ? Modify resource-definition
  Stack trace:

Call backtrace:

  Method                                   Native Class:Line number
  handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:337

Suppressed exception 1 of 1:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337

Error message:
Error has been observed at the following site(s):
  |_ checkpoint ��� Modify resource-definition
Stack trace:

Error context:
  (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.

Call backtrace:

  Method                                   Native Class:Line number
  handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:337
  handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:284
  doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:235
  lambda$doProcessMessage$3                N      com.linbit.linstor.proto.CommonMessageProcessor:220
  subscribe                                N      reactor.core.publisher.FluxDefer:46
  subscribe                                N      reactor.core.publisher.Flux:8357
  onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:418
  drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414
  drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679
  onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243
  drainFused                               N      reactor.core.publisher.UnicastProcessor:286
  drain                                    N      reactor.core.publisher.UnicastProcessor:329
  onNext                                   N      reactor.core.publisher.UnicastProcessor:408
  next                                     N      reactor.core.publisher.FluxCreate$IgnoreSink:618
  next                                     N      reactor.core.publisher.FluxCreate$SerializedSink:153
  processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:383
  doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:218
  lambda$processMessage$2                  N      com.linbit.linstor.proto.CommonMessageProcessor:164
  onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:177
  runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439
  run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526
  call                                     N      reactor.core.scheduler.WorkerTask:84
  call                                     N      reactor.core.scheduler.WorkerTask:37
  run                                      N      java.util.concurrent.FutureTask:264
  run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
  runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1128
  run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:628
  run                                      N      java.lang.Thread:829


END OF ERROR REPORT.



root@okd-node-18:/# linstor error-reports show 6374F2B0-62487-000017
root@okd-node-18:/#

root@okd-node-18:/# linstor r l -r pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
╭────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════╡
╰────────────────────────────────────────────────────────────────╯
root@okd-node-18:/#

root@okd-node-18:/# linstor rd l -r pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Port ┊ ResourceGroup                           ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8 ┊ 7043 ┊ sc-8c6a2bce-347e-5b46-92df-0ac4616314d8 ┊ ok    ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
root@okd-node-18:/#

root@okd-node-18:/# drbdadm status pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8 role:Secondary
  okd-node-05 connection:Connecting
  okd-node-10 connection:Connecting
  okd-node-19 role:Secondary