Kafka Sarama client, optimize by not loading all metadata

[famarting]

Describe the feature

The kafka client used for the kafka components implementation by default has a feature that loads all the topics metadata on initialization, and keeps in memory the metadata for all the topics in the kafka cluster. This is inefficient and memory hungry in scenarios where the kafka cluster used has many topics.

The config property can be found here https://github.com/IBM/sarama/blob/main/config.go#L157

I suggest Dapr always disables that feature in the Sarama client

Release Note

RELEASE NOTE: UPDATE Optimize kafka components for clusters with many topics

RELEASE NOTE:

[famarting]

not applicable

[olitomlinson]

I think its worth re-opening this issue.

I'm not entirely sure why the associated PR was closed tbh, it could be viable!

For context : we have a use case where thousands of partitions on the Kafka cluster are related to Kafka Topics that are never going to be used by Dapr sidecars. So having Dapr load all this metadata on init and refresh it is highly wasteful.

cc @artursouza

[famarting]

I closed the issue because I realized the changes needed to make it work were much larger than what I anticipated...

My hope was that by simply changing a flag, sarama would lazyly read the topic metadata as they are used by produce or consume calls... however that was not the case since I discovered the hard way that sarama requires you to pass the list of topics you'll want to use, maybe after providing an initial list sarama can lazy load new topics... but I didnt get to that point

The issue could be reopened or a new one could be created with a more specific requirement or at least after some more research on what can be done with sarama in terms of optimizing loading the topics metadata

[famarting]

I'm playing a bit with the sarama client again and I think I was suffering from this two issues

IBM/sarama#2630
IBM/sarama#2069

[famarting]

this has not actually been addressed, I found a way of fixing this, it will require a contribution to the sarama client first

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[famarting]

in case anyone sees this, I already have all the code in place but first we need my contribution here to be accepted and released IBM/sarama#2830