Acquiring lock for Azure CosmosDB using Cassandra API not working

[marshall-lamb]

For Azure's CosmosDB emulating Cassandra APIs, there is some odd, as-of-yet unexplained behavior where updates to a table are not immediately available to query, potentially having to do with quorum data consistency. This manifests itself as failure for liquibase to acquire the lock in the DATABASECHANGELOGLOCK table even though queries of the table clearly show the lock was acquired by the liquibase process.

In the 4.23.1 version of this library, in src/main/java/liquibase/ext/cassandra/lockservice/LockServiceCassandra.java, a method was added (isLockedbyCurrentInstance()) that adds a check to see if the lock is held by the current process (by hostname and IP address) that requested it. This seems to have been added to address the behavior observed. This almost works for Azure CosmosDB, however the very next call to acquireLock() from the next changeset operation fails because isLocked() returns true.

To address the problem, I added a call to isLockedbyCurrentInstance() to method isLocked() so that on each iteration of the check for the presence of the lock, it also checks to see if the lock is held by the current process. This resolved the issue.

A separate PR will be created and linked to this issue for review of the actual change.

[marshall-lamb]

Would you please add my ID to authorization to create pull requests? I'm getting denied for marshall-lamb.

[dinchand]

I encounter the same issue. @marshall-lamb could you may be share your PR ?

[marshall-lamb]

As an optimization, I'm removing the return call to executor.queryForInt() as the call to isLockedByCurrentInstance() should be sufficient to indicate whether or not the current process holds the lock.

[marshall-lamb]

@dinchand I'm trying to submit a PR, but I'm getting a 403 access denied error on the push. So far nothing I've tried fixes that.

@r2-lf Can you help me with this? Do I need permissions to push added for my ID?

[giejay]

@marshall-lamb You are not able to create a PR from your forked repo to this one?

[marshall-lamb]

@giejay I got it working. Thanks.

@dinchand Here is the PR: #229

[dinchand]

@marshall-lamb Do you have liquibase working with cosmos(Cassandra) ? I tried but still facing issues...If you have it could you may be share an example ?

[giejay]

I'm using the 4.25 with your fix, and after spending some time on fixing SSL errors, I'm now getting these errors:

Unexpected error running Liquibase: Error executing SQL SELECT TABLE_NAME, KEYSPACE_NAME, COMMENT FROM system_schema.tables WHERE keyspace_name = 'mykeyspace' AND TABLE_NAME = 'databasechangelog': com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query

@marshall-lamb You did not face this issue? There is another blocking issue in the 4.25 release: #242, but I fixed that class cast exception locally.

[maximevw]

Hello @giejay,
I discussed a similar issue with @dinchand here: ing-bank/cassandra-jdbc-wrapper#37 (reply in thread)
The NoNodeAvailableException can have various origins but one of most common is a missing or invalid localdatacenter property when the default load balancing policy is used.

[nicobte]

Hello folks, is there currently a way to disable the database changelog lock? I've been trying to use liquibase-nochangeloglock without success until now. @maximevw @giejay @dinchand

[giejay]

What's the error you are facing? Locking should work now with the newest liquibase Cassandra jar

…

On Tue, Nov 21, 2023, at 7:14 PM, Maxime Wiewiora wrote: Hello @giejay <https://github.com/giejay>, I discussed a similar issue with @dinchand <https://github.com/dinchand> here: ing-bank/cassandra-jdbc-wrapper#37 (reply in thread) <ing-bank/cassandra-jdbc-wrapper#37 (reply in thread)> The `NoNodeAvailableException` can have various origins but one of most common is a missing or invalid `localdatacenter` property when the default load balancing policy is used. — Reply to this email directly, view it on GitHub <#214 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPLZJM7F4SUA325YVR4CFDYFTVPBAVCNFSM6AAAAAA5PQ2GD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRRGQZDEMBQHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[nicobte]

It's not a error, what happens is that I'm running Liquibase updates every time the pod is starting, so when I have to scale many pods at the same time they all have to wait for the lock to be released and that increases the startup time a lot.

I was thinking of a couple of alternatives instead of always acquiring the lock:

1. Acquire the lock only if there are pending changes to execute.
2. Disable database changelog locks.

To achieve this I would have to implement another LockService, right? What do you think? I really appreciate any suggestions.

[maximevw]

Hello @nicobte

Is it relevant to run Liquibase scripts for each pod if they all use the same database? Maybe the good solution should be to run Liquibase separately: execute the updates before deploying your pods. Typically, with K8s, initContainers could be used for that.

However, it's surprising the liquibase-nochangeloglock didn't work, do you have any clue why it fails? Is there any error?

[filipelautert]

nochangeloglock extension should work for that.. Just notice that you need to set parameter databaseChangelogLockEnabled to true.