Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for deterministic listener ports (based on broker ID) #183

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

justinrlee
Copy link

@justinrlee justinrlee commented Jan 8, 2025

This is an enhancement to the dynamic listeners capability, which adds this functionality:

  • Is enabled via the --deterministic-listeners flag
  • When enabled (with dynamic listeners), use broker ID + minimum port to determine listening port (this means that if multiple instances are run in front of a Kafka cluster, all instances will use the same listening ports)
  • Maintain a mapping of broker IDs to broker addresses (and net.Listener objects)
  • If we see a broker with the same broker ID, but a different upstream listener, close the net.Listener and create a new one pointing at the new upstream listener

This is built to support clusters where broker hostnames and ports are not deterministic. Specifically, multiple Confluent Cloud cluster types re-use broker IDs with randomly generated hostnames; this PR enables the Kafka-Proxy to work with these Confluent Cloud clusters.

@everesio any thoughts on this? Would love your input here!

@justinrlee justinrlee changed the title Add support for deterministic listener ports (based on broker ID) feat: Add support for deterministic listener ports (based on broker ID) Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant