kv (ticdc): fix kvClient reconnection downhill loop (#10559)

[ti-chi-bot]

This is an automated cherry-pick of #10559

What problem does this PR solve?

Issue Number: close #10239

This pull request also contain changes of #10674

What is changed and how it works?

1. Add an ID to the eventFeedStream struct to identify a stream and prevent it from being deleted unexpectedly.
2. Bind the cancel function of a gRPC stream to its eventFeedStream to prevent the stream from being canceled unexpectedly.
3. Reduce the number of calls to s.deleteStream to only once to prevent the stream from being deleted unexpectedly.

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. Add hard code time.sleep(5 * time.second) before calling s.deleteStream in both unfixed and fixed version of code.
2. Deploy a TiDB cluster with 3 TiKV nodes and 1 TiCDC server, create a changefeed.
3. Patch the changed cdc binary, and restart a TiKV node randomly and observe the changefeed's lag.

unfixed cdc:

fixed cdc:

From the above graphs, it is evident that in the unfixed CDC, the lag of resolvedTs can exceed 12 minutes when a TiKV node is restarted. However, in the fixed CDC, the increase in resolvedTs is limited to a maximum of 35 seconds. This demonstrates the effectiveness of the fix.

Moreover, when the hard-coded time.sleep(5 * time.second) is removed and the fixed version of CDC is tested again, the lag becomes even smaller:

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug in kv client that could cause an increase in changefeed lag when TiKV is restarted.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sdojjy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sdojjy]

/retest-required