Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of FateLock code for the case when there are lots of locks. #5181

Open
keith-turner opened this issue Dec 13, 2024 · 0 comments
Labels
enhancement This issue describes a new feature, improvement, or optimization.
Milestone

Comments

@keith-turner
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Attempted to intiate many compaction for many small ranges of a table. As more compactions were started the fate operations for each took longer and longer to get a FateLock. Looking at jstacks the code was here and called this and would spend time there. Each fate operation that was added would make the getData calls for all others that had been added before.

Describe the solution you'd like

Attempt to avoid the calls to getData by encoding the information in the node name. The information stored in the data for the node is a fate uuid and whether its a read or write lock. This information could be encoded in the node name. Then the single RPC to zookeeper to getChildren would get call needed information. This would avoid making the RPC for each child to getData.

#5180 will improve the situation somewhat, but there could still be a lot of RPCs.

Additional context

Not sure if this would be suitable for an IT, but would may be good to try. Could try to make an IT for this creates a table w/ 1K tablets and then spins up 1K compactions via API one per tablet. Can try running this before and after the change and looking at the CompactRange.isReady times. Could try increasing 1K also and looking at the time differences as that increases.

@keith-turner keith-turner added the enhancement This issue describes a new feature, improvement, or optimization. label Dec 13, 2024
@keith-turner keith-turner added this to the 4.0.0 milestone Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue describes a new feature, improvement, or optimization.
Projects
None yet
Development

No branches or pull requests

1 participant