Allow processing different partitions concurrently

[dasch]

While messages from a single partition must be processed in-order, messages from different partitions need not. Therefore, we can use multiple threads to process messages from different partitions.

[dasch]

I'm hoping that we can leave the consumer API as-is, and just have this be a configuration. Since both process and process_batch work on single-partition data, we should be able to spin up n independent consumer class instances and have n threads deliver messages to the processing methods independently.

[apeacock1991]

I'm guessing no, but would this parallelise a consumer that subscribes to two topics, each with a single partition?

I was hoping if you had 2 workers, it would assign one topic to one worker, and one topic to another worker 😅