You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During the operation of KeyDB in multi-master mode, a failure occurs where memory consumption increases significantly and network traffic grows. When multi-master-no-forward: yes is enabled, the traffic in normal mode decreases, but during the failure, it also increases. I have noticed that this behavior occurs more frequently if the connection is lost for some time.
To reproduce
Deploy replicas in multi-master mode (in my case, 6 replicas).
Populate with approximately 4GB of data.
Disconnect 1 replica and make data changes on the working replicas.
Restore the disconnected replica.
Wait."
Expected behavior
Expected behavior: data synchronizes.
Actual behavior: a race condition starts with a significant increase in memory consumption.
Additional information
keydb.conf:
protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
supervised no
pidfile /var/run/keydb_6379.pid
loglevel notice
databases 16
always-show-logo yes
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
rdb-del-sync-files no
dir /data
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 100
acllog-max-len 128
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
server-threads 2
replica-weighting-factor 2
active-client-balancing yes
I also overwrite some parameters using launch attributes:
--active-replica yes
-- maxmemory 4gb
-- maxmemory-policy allkeys-lru
-- multi-master-no-forward yes
-- client-output-buffer-limit normal 0 0 0
-- client-output-buffer-limit replica 0 0 0
The text was updated successfully, but these errors were encountered:
Danozavr
changed the title
[BUG]
[BUG] Incorrect memory usage in multi-master mode
Jun 21, 2024
During the operation of KeyDB in multi-master mode, a failure occurs where memory consumption increases significantly and network traffic grows. When
multi-master-no-forward: yes
is enabled, the traffic in normal mode decreases, but during the failure, it also increases. I have noticed that this behavior occurs more frequently if the connection is lost for some time.To reproduce
Expected behavior
Expected behavior: data synchronizes.
Actual behavior: a race condition starts with a significant increase in memory consumption.
Additional information
keydb.conf:
protected-mode noport 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
supervised no
pidfile /var/run/keydb_6379.pid
loglevel notice
databases 16
always-show-logo yes
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
rdb-del-sync-files no
dir /data
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 100
acllog-max-len 128
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
server-threads 2
replica-weighting-factor 2
active-client-balancing yes
I also overwrite some parameters using launch attributes:
![image](https://private-user-images.githubusercontent.com/127389790/340323881-579e695f-7cde-44d2-860f-1d3c84aab068.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE5NzU0MDEsIm5iZiI6MTcyMTk3NTEwMSwicGF0aCI6Ii8xMjczODk3OTAvMzQwMzIzODgxLTU3OWU2OTVmLTdjZGUtNDRkMi04NjBmLTFkM2M4NGFhYjA2OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyNlQwNjI1MDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02YjExYTFhNzJjMGUyZjQzZWI0YjUyOGYxM2ExZmQ4YzEzYjZlNjNlOTY1MmQyZDA3ZDk5MmE5NTg0ZGYzMmY2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.L_kckVv_Yv9-Hjne-ryhUALHGNoeb78iGSH1Uc5V8o8)
![image](https://private-user-images.githubusercontent.com/127389790/340324040-cfa993b8-bb54-474c-b681-f77a1c6ab114.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE5NzU0MDEsIm5iZiI6MTcyMTk3NTEwMSwicGF0aCI6Ii8xMjczODk3OTAvMzQwMzI0MDQwLWNmYTk5M2I4LWJiNTQtNDc0Yy1iNjgxLWY3N2ExYzZhYjExNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyNlQwNjI1MDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zOTJkNjNmZjRjZTA5M2RjZGFkOTIyZTA1NzE5OWFmZDQ5M2RiZDNiY2I2MzI4NzM3YTA5ZTdmZWJlMjFjMjU4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.4Q5GFdtBGT4D8zIcE-jTVR_B23CG-vNhNJxAVbMaTOM)
--active-replica yes
-- maxmemory 4gb
-- maxmemory-policy allkeys-lru
-- multi-master-no-forward yes
-- client-output-buffer-limit normal 0 0 0
-- client-output-buffer-limit replica 0 0 0
The text was updated successfully, but these errors were encountered: