Rule mysteriously stops running until elastalert service restarted

[krivard]

Hi, we've got a half dozen elastalert rules set to run once a minute, and occasionally one or more of them will just... not run anymore. It picks back up when we restart the elastalert service with nary a peep.

This is a major problem for rules that aren't expected to alert very often, since we don't have a good way to tell if everything's fine or if the alerter just isn't running. Thankfully the most recent time this happened, it was on a "everything's good" alert that we expect to see every day, so I was able to catch the below in syslog just after the most recent successful run of the rule.

My questions are:

• How can I prevent this from happening? or if not,
• How can I monitor the system in such a way that I hear about it whenever it happens?

Jul 30 18:39:08 python[3350506]: WARNING:elasticsearch:DELETE http://[redacted].cmu.edu:9200/_search/scroll [status:429 request:0.002s]
Jul 30 18:39:08 python[3350506]: ERROR:root:Traceback (most recent call last):
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 1269, in handle_rule_execution
Jul 30 18:39:08 python[3350506]:     num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 905, in run_rule
Jul 30 18:39:08 python[3350506]:     if not self.run_query(rule, rule['starttime'], endtime):
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 663, in run_query
Jul 30 18:39:08 python[3350506]:     self.thread_data.current_es.clear_scroll(scroll_id=scroll_id)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
Jul 30 18:39:08 python[3350506]:     return func(*args, params=params, **kwargs)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 1290, in clear_scroll
Jul 30 18:39:08 python[3350506]:     "DELETE", "/_search/scroll", params=params, body=body
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/transport.py", line 318, in perform_request
Jul 30 18:39:08 python[3350506]:     status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
Jul 30 18:39:08 python[3350506]:     self._raise_error(response.status_code, raw_data)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
Jul 30 18:39:08 python[3350506]:     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
Jul 30 18:39:08 python[3350506]: elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [3975118756/3.7gb], which is larger than the limit of [3950719795/3.6gb], real usage: [3975118328/3.7gb], new bytes reserved: [428/428b], usages [request=248640/242.8kb, fielddata=92569280/88.2mb, in_flight_requests=34800/33.9kb, model_inference=0/0b, accounting=136139674/129.8mb]')
Jul 30 18:39:08 python[3350506]: ERROR:root:Uncaught exception running rule Completed indicator run: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [3975118756/3.7gb], which is larger than the limit of [3950719795/3.6gb], real usage: [3975118328/3.7gb], new bytes reserved: [428/428b], usages [request=248640/242.8kb, fielddata=92569280/88.2mb, in_flight_requests=34800/33.9kb, model_inference=0/0b, accounting=136139674/129.8mb]')
Jul 30 18:39:08 python[3350506]: WARNING:elasticsearch:POST http://logs-prod-01.delphi.cmu.edu:9200/elastalert_status_error/_doc [status:429 request:0.002s]
Jul 30 18:39:08 python[3350506]: ERROR:root:Error writing alert info to Elasticsearch: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [3993998072/3.7gb], which is larger than the limit of [3950719795/3.6gb], real usage: [3993992696/3.7gb], new bytes reserved: [5376/5.2kb], usages [request=248640/242.8kb, fielddata=92569280/88.2mb, in_flight_requests=39748/38.8kb, model_inference=0/0b, accounting=136139674/129.8mb]')
Jul 30 18:39:08 python[3350506]: Traceback (most recent call last):
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 1269, in handle_rule_execution
Jul 30 18:39:08 python[3350506]:     num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 905, in run_rule
Jul 30 18:39:08 python[3350506]:     if not self.run_query(rule, rule['starttime'], endtime):
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 663, in run_query
Jul 30 18:39:08 python[3350506]:     self.thread_data.current_es.clear_scroll(scroll_id=scroll_id)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
Jul 30 18:39:08 python[3350506]:     return func(*args, params=params, **kwargs)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 1290, in clear_scroll
Jul 30 18:39:08 python[3350506]:     "DELETE", "/_search/scroll", params=params, body=body
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/transport.py", line 318, in perform_request
Jul 30 18:39:08 python[3350506]:     status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
Jul 30 18:39:08 python[3350506]:     self._raise_error(response.status_code, raw_data)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
Jul 30 18:39:08 python[3350506]:     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
Jul 30 18:39:08 python[3350506]: elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [3975118756/3.7gb], which is larger than the limit of [3950719795/3.6gb], real usage: [3975118328/3.7gb], new bytes reserved: [428/428b], usages [request=248640/242.8kb, fielddata=92569280/88.2mb, in_flight_requests=34800/33.9kb, model_inference=0/0b, accounting=136139674/129.8mb]')
Jul 30 18:39:08 python[3350506]: During handling of the above exception, another exception occurred:
Jul 30 18:39:08 python[3350506]: Traceback (most recent call last):
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elastalert/elastalert.py", line 1614, in writeback
Jul 30 18:39:08 python[3350506]:     res = self.writeback_es.index(index=index, body=body)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
Jul 30 18:39:08 python[3350506]:     return func(*args, params=params, **kwargs)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 362, in index
Jul 30 18:39:08 python[3350506]:     body=body,
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/transport.py", line 318, in perform_request
Jul 30 18:39:08 python[3350506]:     status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
Jul 30 18:39:08 python[3350506]:     self._raise_error(response.status_code, raw_data)
Jul 30 18:39:08 python[3350506]:   File "/root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
Jul 30 18:39:08 python[3350506]:     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
Jul 30 18:39:08 python[3350506]: elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [3993998072/3.7gb], which is larger than the limit of [3950719795/3.6gb], real usage: [3993992696/3.7gb], new bytes reserved: [5376/5.2kb], usages [request=248640/242.8kb, fielddata=92569280/88.2mb, in_flight_requests=39748/38.8kb, model_inference=0/0b, accounting=136139674/129.8mb]')

[nsano-rururu]

I think elastalert2 requires python 3.9, why python 3.6? .. I feel that there was a function that did not work properly with python3.6.

[nsano-rururu]

It looks like I'm getting a circuit breaker exception indicating that the elasticsearch data is too big, can't I look at the logs to find out the cause?

Error code 429 - circuit_breaking_exception
https://discuss.elastic.co/t/error-code-429-circuit-breaking-exception/202494

[nsano-rururu]

Java heap memory exhausted due to over-opening of elasticsearch index
circuit_breaking_exception may occur
→ ElasticSearch heap size change
→ Elastic Search
Changed indices.breaker.total.limit
→ Delete Index data of ElasticSearch, or close unnecessary Index data

[nsano-rururu]

You should consider using Elasticsearch's Index Lifecycle Management (ILM) to remove indexes that are older than the specified time period.

[nsano-rururu]

If you have enough disk space, you can close the indexes stored in Elasticsearch on a regular basis. Even if you have enough disk space, the Java heap memory has been exhausted and Elasticsearch has become unresponsive. When closed, the index will not be searched and memory usage will decrease.

[krivard]

Thanks! I found a pile of ancient indexes and closed them. I was a bit concerned by some of the Elastic docs which say that closing indexes can sometimes result in data loss, but it turns out that's only for cloud deployments, and we have a single-node self-hosted setup.

Here's hoping that does the trick! 🤞