feat: Yield inside huge values migration serialization #4197
Conversation
|
@adiholden re/ locks during serialization:
|
don't worry about |
I do refer to |
| if seed_during_migration: | ||
| await stop_seed() | ||
| else: | ||
| # Only verify memory growth if we haven't pushed new data during migration |
There was a problem hiding this comment.
I am not sure we need this option of not running seeder during migration
you added this option inorder to check the rss compared to rss before migration right? but we can compare peak rss to peak used memory
tests/dragonfly/cluster_test.py
Outdated
|
|
||
| insert_task = asyncio.create_task(insert_data(instances[0].cluster_client())) | ||
|
|
||
| async def get_rss(client, field): |
There was a problem hiding this comment.
get_meomory_info maybe?
tests/dragonfly/cluster_test.py
Outdated
| while True: | ||
| rss = await get_rss(nodes[0].client, "used_memory_rss") | ||
| logging.debug(f"Current rss: {rss}") | ||
| if rss > 1_000_000_000: |
There was a problem hiding this comment.
explain why are you waiting for 1G rss?
tests/dragonfly/cluster_test.py
Outdated
| # Insert data to containers with a gaussian distribution: some will be small and other big | ||
| stop = False | ||
|
|
||
| async def insert_data(client): |
There was a problem hiding this comment.
Kostas is working on seeder improvement for containers with different size
I prefer to have something generic and this logic inside test
|
Please do not review yet. I still haven't modified the tests to use Kostas' new framework. This is just a merge of changes from |
With #4144 we break huge values slot migration into multiple commands. This PR now adds yield between those commands.
It also adds a test that checks that modifying huge values while doing a migration works well, and that RSS doesn't grow too much.
Fixes #4100