feat(sensor): disable leader election for jetstream eventbus

[ryancurrah]

Supersedes #3240

Checklist:

• My organization is added to USERS.md.

[whynowy]

What is the reason leader election is not needed when using jetstream? Assume the trigger relies on events from multiple sources, with multiple sensor instances, it will be an issue right?

[ryancurrah]

Leader election is not needed when using JetStream because, like Kafka consumer groups, it supports horizontal scaling through queue subscriptions, which distribute events across multiple sensor instances, ensuring that only one instance processes each event. Similar to Kafka consumer groups, where each message is processed by only one consumer within a group, JetStream queue subscriptions achieve the same event distribution. It is important to ensure that queue subscriptions are enabled; otherwise, if sensor leader election is turned off, all sensor instances will receive and process the same events, leading to duplicate event processing.

Before disabling leader election for JetStream, my PR should verify that queue subscriptions are enabled to ensure that events are handled by only one sensor instance.

Queue subscription support was added in this commit: 5461296.

[whynowy]

Leader election is not needed when using JetStream because, like Kafka consumer groups, it supports horizontal scaling through queue subscriptions, which distribute events across multiple sensor instances, ensuring that only one instance processes each event. Similar to Kafka consumer groups, where each message is processed by only one consumer within a group, JetStream queue subscriptions achieve the same event distribution. It is important to ensure that queue subscriptions are enabled; otherwise, if sensor leader election is turned off, all sensor instances will receive and process the same events, leading to duplicate event processing.

Before disabling leader election for JetStream, my PR should verify that queue subscriptions are enabled to ensure that events are handled by only one sensor instance.

Queue subscription support was added in this commit: 5461296.

This is not my concern. My concern is about the following scenario:

Sensor s1 has dependencies on es1 && es2, with multiple sensor instances, those events can go to different pods, which will not be able to meet the && condition.

[ryancurrah]

This is not my concern. My concern is about the following scenario:

Sensor s1 has dependencies on es1 && es2, with multiple sensor instances, those events can go to different pods, which will not be able to meet the && condition.

Hrm how does it work for Kafka? I assume the concerns would have also applied to Kafka, and if so how did you solve it for Kafka?

This is the PR in which Kafka was added 95a0877#diff-57d9dbd85ed75b31a65e6880e99ee6ae77344b1760efa44ad73e82a2dce287bcR46

[mweibel]

so this was closed but my earlier PR related to this (#3240) is still open. Any chance this could go further?