Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safekeepers should gossip remote_consistent_lsn via broker #8148

Open
petuhovskiy opened this issue Jun 24, 2024 · 1 comment
Open

Safekeepers should gossip remote_consistent_lsn via broker #8148

petuhovskiy opened this issue Jun 24, 2024 · 1 comment
Labels
c/storage/safekeeper Component: storage: safekeeper t/bug Issue Type: Bug triaged bugs that were already triaged

Comments

@petuhovskiy
Copy link
Member

I saw this happenning in tests:

  1. pageserver updates remote_consistent_lsn to match last_record_lsn
  2. connected safekeeper receives new LSN
  3. safekeeper updates broker_is_active to false
  4. it stops pushing updates to broker
  5. other safekeepers have no chance to learn new remote_consistent_lsn from broker

The fix is to delay timeline deactivation for some time (30s), so that safekeepers would have a chance to broadcast remote_consistent_lsn update to peers. It's not a solution for 100% of cases, but should work good enough.

@petuhovskiy petuhovskiy added t/bug Issue Type: Bug c/storage/safekeeper Component: storage: safekeeper labels Jun 24, 2024
@jcsp
Copy link
Contributor

jcsp commented Jun 26, 2024

(notes chatting with Arthur)

Impact: interferes with writing clean tests. Currently if a safekeeper has stale remote_consistent_lsn for long enough, it will remain active & the pageserver will eventually connect to it. When the pageserver connects it will eventually learn remote_consistent_lsn.

More generally: should we reconsider using remote_consistent_lsn in the safekeeper in our condition for broker_is_active?

@jcsp jcsp added the triaged bugs that were already triaged label Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/safekeeper Component: storage: safekeeper t/bug Issue Type: Bug triaged bugs that were already triaged
Projects
None yet
Development

No branches or pull requests

2 participants