[Enhancement]: Handle response with status 409 when talking to KafkaConnect API

[tinaselenge]

Related problem

KafkaConnectAPIImpl class has the following TODO comment in a few places:

// TODO Handle 409 (Conflict) indicating a rebalance in progress

Suggested solution

We should investigate if it still needs to be handled or not. If yes, implement handling of this response and these TODO comments should be removed either way.

Alternatives

No response

Additional context

No response

[ppatierno]

Triaged on 12/12/2024: agreed to take this open and investigate if this needs to handled. @tinaselenge volunteered to take a look at it. Thanks Tina!

[tinaselenge]

After doing some investigation, it looks like 409 can be thrown with the following endpoints:

• POST /connectors (creating a connector)
• PUT /connectors/{name}/config (updating a connector configuration)
• DELETE /connectors/{name} (deleting a connector)

Modifying, updating or deleting a connector resource may require the request to be forwarded to the leader and Connect will return HTTP 409 while the worker group rebalance is in process as the leader may change during rebalance.

The suggested way to handle this is to retry the request with a delay. We already do this in our Connect API client implementation using BackOff for some of the requests. I will raise a PR to change requests for these endpoints to have the same retry logic for 409.

Kafka doc suggests that restarting a connector or task could also return 409. However, according to the the following statement, retry for this request is not necessary:

You can retry after the rebalance completes, but it might not be necessary since rebalances effectively restart all the connectors and tasks in the cluster.

Sources:
https://kafka.apache.org/documentation/#connect_administration
https://docs.confluent.io/platform/current/connect/references/restapi.html