Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unexpected election when one of followers is network partition with leader #9020

Open
lhy1024 opened this issue Jan 26, 2025 · 0 comments · May be fixed by #9021
Open

unexpected election when one of followers is network partition with leader #9020

lhy1024 opened this issue Jan 26, 2025 · 0 comments · May be fixed by #9021
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jan 26, 2025

Enhancement Task

An unexpected election occurs when one of the followers experiences a network partition from the leader.

If the etcd client is using a follower endpoint and encounters a failure when attempting to save a timestamp, the allocator_manager will reset the leader.

[2025/01/21 20:39:49.149 +08:00] [WARN] [lease.go:187] ["lease keep alive failed"] [purpose="leader election"] [start=2025/01/21 20:39:46.149 +08:00] [error="context deadline exceeded"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [etcd_kv.go:180] ["txn runs too slow"] [response=null] [cost=5.721552451s] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [tso.go:333] ["save timestamp failed"] [] [timestamp-path=timestamp] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [allocator_manager.go:289] ["failed to update allocator's timestamp"] [] [name=tc-pd-1] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]

In fact, manual retry is needed, next request succeeds with the new endpoint, according to etcd-io/etcd#8711
@lhy1024 lhy1024 added the type/enhancement The issue or PR belongs to an enhancement. label Jan 26, 2025
@lhy1024 lhy1024 linked a pull request Jan 26, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tso: add retry for UpdateTSO lhy1024/pd
1 participant
@lhy1024