Memory consumption of pipeline

[kushsharma]

I was looking at the code to understand how the source to sink pipeline works and one thing I noticed is each source keeps on appending new Part object to Message and in Get or Iter of Message we are not really deleting this. Am I missing something or do we really want these messages to pile up in memory even after they are sunk?
Great work by the way! 😄

[Jeffail]

Hey @kushsharma, that's definitely not an intended behaviour, where exactly are you seeing that?

[kushsharma]

@Jeffail I could be wrong as I only briefly went through the code. Let's take the example of Kafka In and Kafka Out.
Kafka reader appends message parts here, which then gets consumed by Kafka reader here over an iterator. Now if I look at iterator code, it is simply executing the function.
I want to understand does size of a single types.Message defines the batch size? Does the core calls kafka reader multiple times for Msg and passes it to the writer? Or it's a one-time thing?

[Jeffail]

Hey @kushsharma, the internal code base is a bit odd in that types.Message is actually a batch of one or more messages, so what's being iterated is a single instance of a batch of messages. A new batch is created for each call to consume so there's no unbounded growth in that call.