Adding placementCount to an existing volume

[Ulrar]

Hi,

Apologies if this is documented somewhere and I missed it, I've never used this before.

I have a one node Talos cluster (controlplane and worker at the same time) at home and I wonder what the behavior would be if I followed the tutorial and created a class without a placementCount at first, then added a couple of more Talos nodes in the future and bumped the placementCount up to 2 or 3 then.

Would this be supported, or would it require creating a new one and migrating existing data ?

Thanks !

[WanzenBug]

There is a slight issue, in that the StorageClass can't simply be updated. This is not allowed by Kubernetes.

What you can do is delete the StorageClass and then recreate it with updated Parameters. But that will not automatically change the already deployed volumes, it will only affect new volumes.

You can do the following:

• First create a StorageClass which sets no placementCount, but does set a resourceGroup, for example to the same name as the StorageClass
• After adding new nodes, delete and recreate the StorageClass, setting placementCount and keeping the same resourceGroup name.
• Run kubectl exec deploy/linstor-controller -- linstor resource-group modify --place-count 2 <resourceGroupName> to update the existing volumes

[Ulrar]

Hi,

Thanks, I decided not to tempt fate and just test this using a VM as a temporary node, and your instructions did seem to work.

Although when I ran the last command to update the place count, my cluster became unavailable for a few minutes, it looks like it was so heavy on the drives of that VM that etcd started timimg out and since I only had two nodes at the time, the whole thing went down until the sync completed. I guess it's not directly related to this project but good learning for me, clearly.

Thanks again !

[Ulrar]

Actually one last question, after a bit more testing I added a third node and it doesn't seem to be adding the expected TieBreaker / diskless volumes on it, presumably because when I bumped the placementCount this wasn't possible.

Is there a way to force it now ? I tried playing with diskless-on-remaining but that didn't seem to do anything, and re-setting placement count to 2 also doesn't do anything

EDIT: Nevermind, the node lists as online in linstor but it's actually not, the satellite is crashing because DRBD is missing. Looks like the DRBD extension on arm64 isn't working the same as on amd64 on Talos, that's likely the reason here

[WanzenBug]

Hmm, yes, initial sync can create a lot of load if you are not careful.

[Ulrar]

I got a second physical node, so I evacuated the VM node then restored it. The volumes did move to the new node as expected, but it's not re-creating TieBreakers on the now restored vm node. Is there a trick to make it reconsider the node as a valid target to use for tie breaking ?

Side note for anyone else stumbling upon this, I followed the linstor doc on evacuating a node which says to drain the node and that killed the DRBD pod on it (it's not part of a daemon set so it wasn't excluded), which then led to the node being tainted with drbd.linbit.com/lost-quorum:NoSchedule and broke all the volumes. Thankfully untaiting it manually allowed the pod to restart and the volumes recovered automatically, since all the pods using them were dead no changes were made during that time.