About mail sending function

[ma-anazawa]

I would like to be notified by email when "NRTM update" or "complete import by downloading from FTP site" fails.
When "NRTM update" or "Full import by downloading from FTP site" failed, the following message was logged.
Therefore, I am thinking of sending an email when'irrd.mirroring.mirror_runners_import # ERROR'or'% ERROR:' is detected in the log.
Is there any other good implementation method?

■ Example of message logged when NRTM update fails

2021-07-27 04:48:58,503 irrd[1852364]: [irrd.mirroring.mirror_runners_import#DEBUG] Received NRTM response for JPIRR: % ERROR: mirror access denied
2021-07-27 05:48:23,682 irrd[1247221]: [irrd.mirroring.mirror_runners_import#DEBUG] Received NRTM response for RADB: % ERROR: serials (5489854 - 6284368) don't exist!
2020-10-24 23:59:33,012 irrd[3005748]: [irrd.mirroring.mirror_runners_import#DEBUG] Received NRTM response for AOLTW: % ERROR: Unknown db (AOLTW) in mirror request
2021-07-27 22:59:15,302 irrd[2274872]: [irrd.mirroring.mirror_runners_import#ERROR] An error occurred while attempting a mirror update or initial import for ARIN: [Errno -2] Name or service not known

■ Examples of messages logged when full FTP import fails

2021-07-27 23:59:20,554 irrd[2277073]: [irrd.mirroring.mirror_runners_import#ERROR] An error occurred while attempting a mirror update or initial import for APNIC: [Errno -2] Name or service not known
2021-07-27 23:16:13,436 irrd[1285653]: [irrd.mirroring.mirror_runners_import#ERROR] An exception occurred while attempting a mirror update or initial import for APNIC: 550 /pub/apnic/whois/1apnic.db.as-set.gz: No such file or directory

[mxsasha]

This is probably possible, but non-trivial, with custom logging configuration: https://irrd.readthedocs.io/en/stable/admins/configuration/#logging
You'd need to dive into the Python logging framework for this, to have a custom logger that detects the cases you want to log, and then set up a handler that sends emails. Also, be aware that as the docs say, misconfigurations can lead to log messages being silently lost.

Error responses from NRTMv3 queries are also ambiguous, which is why IRRD logs them at DEBUG level. Failed file downloads or TCP connections are clear errors, hence they are logged as ERROR. The way NRTMv3 works in IRRD is that we have data up to serial X, and request all updates from X-1 until LAST. If X+1 does not exist yet, the server may return an error. So where % ERROR: mirror access denied is pretty clear, errors about serials not existing could mean that your IRRD is too far behind, or that it is entirely up to date and trying to retrieve new data, when there is nothing new yet. Unfortunately, NRTMv3 does not have a structured way to distinguish this, so IRRD's method is to just try and keep trying.

[ma-anazawa]

Thank you for the information.
I would like to consider a little more.

[mxsasha]

Oh, one more thing, @ma-anazawa. Monitoring the logs is generally just not the best approach for monitoring mirror status. For one, because NRTM error messages are not very clear, but also because it might not catch all issues. For a better method, see the !J and !j queries, which you can run against your own instance and the instance that you're mirroring from. This will catch all kinds of errors, and then you can use the log file to figure out what happened.