-
Notifications
You must be signed in to change notification settings - Fork 968
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(server): fix compatibility with rdb snapshot #3121
Conversation
Signed-off-by: adi_holden <[email protected]>
c_master = aioredis.Redis(port=redis_local_server.port) | ||
await c_master.ping() | ||
|
||
assert await c_master.dbsize() == dbsize |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First redis compatibility test 👍🏻
If more checks are needed, you can re-define dragonfly.ihash
in redis. It's bit of a hassle, but doable 🙂
local dragonfly = {}
function draognfly.ihash(hash, sort, ...)
local res = redis.pcall(arg)
-- if sort, sort res
-- use any hash function to res... lua really doesn't have one :(
end
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to use the seeder capture but got error
redis.exceptions.ResponseError: Error compiling script (new function): user_script:1: unexpected symbol near '#'
I want the fix to go to 1.19, so I will merge this now and can investigate more if we can do a simple change to make this work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adiholden you got an error with Redis, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you mean the script error? yes its in redis
Signed-off-by: adi_holden <[email protected]>
…nfly ( v1.18.1 → v1.19.0 ) (#3784) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>
…nfly ( v1.18.1 → v1.19.0 ) (#629) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTAuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM5MC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: kireque-bot[bot] <143391978+kireque-bot[bot]@users.noreply.github.com>
…19.0 ) (#4821) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>dragonflydb/dragonfly (ghcr.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTAuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM5MC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>
The bug: writing full sync cut opcode to rdb snapshot format breaks compatibility with rdb format
PR changes:
Do not write dfly full sync cut opcode for disk snapshot (not for df and not for rdb)
Add pytest for loading rdb snapshot generated by dragonfly with Redis server