Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexpected election when one of followers is network partition with leader #9020

Closed
lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Closed

unexpected election when one of followers is network partition with leader #9020

lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jan 26, 2025

Enhancement Task

An unexpected election occurs when one of the followers experiences a network partition from the leader.

If the etcd client is using a follower endpoint and encounters a failure when attempting to save a timestamp, the allocator_manager will reset the leader.

[2025/01/21 20:39:49.149 +08:00] [WARN] [lease.go:187] ["lease keep alive failed"] [purpose="leader election"] [start=2025/01/21 20:39:46.149 +08:00] [error="context deadline exceeded"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [etcd_kv.go:180] ["txn runs too slow"] [response=null] [cost=5.721552451s] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [tso.go:333] ["save timestamp failed"] [] [timestamp-path=timestamp] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [allocator_manager.go:289] ["failed to update allocator's timestamp"] [] [name=tc-pd-1] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]

In fact, manual retry is needed, next request succeeds with the new endpoint, according to etcd-io/etcd#8711

@lhy1024 lhy1024 added the type/enhancement The issue or PR belongs to an enhancement. label Jan 26, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in #9021 Feb 5, 2025
ti-chi-bot bot added a commit that referenced this issue Feb 5, 2025
close #9020

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@lhy1024 lhy1024 added type/bug The issue is confirmed as a bug. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major and removed type/enhancement The issue or PR belongs to an enhancement. labels Feb 11, 2025
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: lhy1024 <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: lhy1024 <[email protected]>
ti-chi-bot bot added a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: lhy1024 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant