Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider if we should make TiKV the primary source of Region information #55013

Open
MyonKeminta opened this issue Jul 29, 2024 · 0 comments
Open
Labels
type/question The issue belongs to a question.

Comments

@MyonKeminta
Copy link
Contributor

Ref: tikv/client-go#1398

We recently met such an issue:

  • The PD leader has out-of-date region info
  • A region did a conf change and a new replica is added; the new replica becomes the leader
  • The client has the old region info without the conf change in the region cache

Then when the client tries to access the region's leader:

  1. The client tries to access one of the region's old replicas
  2. The TiKV has both mismatching epoch and leader. Due to it's implementation, TiKV returns the NotLeader error prior to EpochNotMatch, carrying the new leader's peer info
  3. The client tries to switch to the new leader, but as it has an old region info without the conf change, the leader is missing in its peer list. It then invalidates the region.
  4. Then it will trigger reloading the region info from PD. But as the information held by the current PD leader is also out of date, it failed to get the valid region information for sending the request until the PD updates the information. This causes the access fails for about two minutes.

A simple solution to this problem would be: carrying the full region information when reporting the NotLeader error, and use it to update the region cache if the returned region info has larger epoch than that in region cache.

But in this issue (tikv/client-go#1398), the problem was more deeply considered. We may need to think about what we can do further about improving the client's region info updating.

@MyonKeminta MyonKeminta added the type/question The issue belongs to a question. label Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/question The issue belongs to a question.
Projects
None yet
Development

No branches or pull requests

1 participant