fix(RedisMessageStore): RedisMessageStore add lock #9680
Conversation
I found that when RedisMessageStore adds and removes messages, it operates on two keys separately, which may cause problems in multi-threading due to non-atomic operations. Although using redis to delay messages is not a good idea, the abnormal loss of messages in the logs alerted me when the number of requests was not large. By comparing the logs, I found the problem that the message group representing the metadata is not consistent with the actual message. A simple solution is to add lock like SimpleMessageStore, which is also the approach taken in this pull request. This will bring some performance loss, and I am not sure whether a configable switch is needed.
There was a problem hiding this comment.
I think this change somehow related to: #5123. At least by the title of the issue: the rest of the discussion went sideways.
Can we also think about moving this locking logic into a super class AbstractKeyValueMessageStore?
This way any impl would benefit immediately.
I also wonder if it is really safe to say that this is not a new feature which should go to the next 6.5 instead of making new bugs from existing...
Thank you for bringing this up!
...ation-redis/src/main/java/org/springframework/integration/redis/store/RedisMessageStore.java
Outdated
Show resolved
Hide resolved
...ation-redis/src/main/java/org/springframework/integration/redis/store/RedisMessageStore.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Well, actually, it sounds more like we have to pull the lock usage up to the AbstractMessageGroupStore. So, all the impls would benefit.
Including that SimpleMessageStore refactoring I have mentioned.
Let me know if you are OK to work on all of that!
Otherwise I'll take it from here.
But again: that is going to be as part of the next 6.5 version.
|
So, I would say it is official: this PR is about to fix #5123. |
|
@artembilan I will be working on this for a week or so. If still cannot meet the pull request by then, please you continue with the remaining work. |
|
No problem , @NaccOll , take your time! We have not planned |
There was a problem hiding this comment.
This is really cool!
Please, consider to mention such an improvement in the message-store.adoc.
I'll come back to you for the whats-new.adoc change request as well, when I switch main to 6.5.0 - somewhere in the end of month or so. When we done with follow-up bug fixes for 6.4.x generation.
Thanks
...n-core/src/main/java/org/springframework/integration/store/AbstractKeyValueMessageStore.java
Outdated
Show resolved
Hide resolved
...n-core/src/main/java/org/springframework/integration/store/AbstractKeyValueMessageStore.java
Outdated
Show resolved
Hide resolved
...tion-core/src/main/java/org/springframework/integration/store/AbstractMessageGroupStore.java
Outdated
Show resolved
Hide resolved
...ation-redis/src/main/java/org/springframework/integration/redis/store/RedisMessageStore.java
Outdated
Show resolved
Hide resolved
|
In addition to the Lock mechanism, SimpleMessageStore also introduces a semaphore mechanism. My code will cause an error in the |
I found that when RedisMessageStore adds and removes messages, it operates on two keys separately, which may cause problems in multi-threading due to non-atomic operations.
Although using redis to delay messages is not a good idea, the abnormal loss of messages in the logs alerted me when the number of requests was not large. By comparing the logs, I found the problem that the message group representing the metadata is not consistent with the actual message.
I discovered this problem at least three years ago, but not many people use redis as a message storage, and there is little feedback from the community. I solved this problem by inheriting RedisMessageStore in my project.
My current solution is to add a lock like SimpleMessageStore, which is also the method used in this pull request. After several years of production environment verification, this is feasible. But this is bound to bring some performance loss, which is why I have not yet initiated a pull request to spring-integrate. But now I think it is troublesome to duplicate this class between multiple projects, and not all project members are aware of this problem, so I hope to solve this problem at the framework level.
like #463