How can I stop benthos when errror occurs?

[artemklevtsov]

Hi,

Can I stop benthos when errors occurs when processing messages (pipelines stage)? For the input I found read_unitl and retry (with count = 1) for the output.

[Jeffail]

Hey @artemklevtsov, unfortunately we don't really have mechanisms for this in benthos as it's a stream processor and stopping the service for any reason is typically an absolute last resort. I'm assuming you're using benthos as more of a batch processor in this case but we can't really offer anything out of the box, you could potentially write a simple processor plugin that calls os.Exit and execute it upon an error, or use the command processor to do something naughty with ps and kill.

[artemklevtsov]

I want to stop process stream when schema suddenly changed and fix it.
May something like exit processor which send OS singal?

[Jeffail]

If you use a reject output like this: https://www.benthos.dev/docs/configuration/error_handling#reject-messages then the failed data will be continuously retried with back-off, this applies back pressure where benthos will stop consuming new data until you fix the problem. This is generally a much safer way to fix configuration issues as it means other data that isn't problematic can potentially still make its way through the system while a solution is found, and it means random errors don't arbitrarily kill the application.

Exiting the program in production also has a lot of negative side effects and puts extra strain on upstream and downstream services that are repeated connected to and disconnected from.