Controlling the number of Retries for negative response from Elastalert in container logs

[kumarabie]

Dear community,
I am getting lots of retries for negative response from container logs such as 502 Bad gateway, which in turn bursting the logs
and also I am using these alerts for the purpose of creating traces which successively overcrowding the traces.

• Is there any way to reduce retries or to control retries from elastalert codebase .

INFO:elastalert:Sleeping for 59.999786 seconds
INFO:elastalert:Queried rule failedrule_serverless from 2021-06-12 10:44 UTC to 2021-06-15 10:44 UTC: 171 / 171 hits
ERROR:root:Error while running alert http_post: Error posting HTTP Post alert: 502 Server Error: Bad Gateway for url: http://failing-func.serverless.svc.cluster.local/?_failedrule_serverless
INFO:elastalert:Ignoring match for silenced rule failedrule_serverless
INFO:elastalert:Ran failedrule_serverless from 2021-06-12 10:44 UTC to 2021-06-15 10:44 UTC: 171 query hits (169 already seen), 2 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2021-06-15 10:45 UTC
ERROR:root:Error while running alert http_post: Error posting HTTP Post alert: 502 Server Error: Bad Gateway for url: http://failing-func.serverless.svc.cluster.local/?_failedrule_serverless
ERROR:root:Error while running alert http_post: Error posting HTTP Post alert: 502 Server Error: Bad Gateway for url: http://failing-func.serverless.svc.cluster.local/?_failedrule_serverless
ERROR:root:Error while running alert http_post: Error posting HTTP Post alert: 502 Server Error: Bad Gateway for url: http://failing-func.serverless.svc.cluster.local/?_failedrule_serverless

Any help is appreciated :-)

[jertel]

I'm not aware of a way to disable failed alerts from retrying. However, if you'd like to submit a PR to add that feature, you are welcome to do so.

[kumarabie]

I haven't gone through the code entirely yet, so if you have any suggestions on adding this feature or pointers on how these retries are even happening, I can try to add this feature.
I suspect that this might be happening because the errors might be stored elastalert_status_error index so it keeps on trying while the error is there. would you happen to have any idea about this?

[nsano-rururu]

ElastAlert Metadata Index
https://elastalert2.readthedocs.io/en/latest/elastalert_status.html

elastalert_status_error
　Error log
elastalert_status_status
　Query log
elastalert_status
　Alert log
elastalert_status_silence
　Silence log

[ferozsalam]

Hey @kumarabie have you tried using the alert_time_limit parameter in your config.yaml? I think you can set it to zero (default is two days), which might solve your issue.

See here for an example of the syntax.

[kumarabie]

@ferozsalam It worked for me.Thanks