Error back channel / side channel

[lars-t-hansen]

The normal case with errors is that sonar starts and produces output, but that some error occurs during the run that prevents it from completing or producing correct data. (The abnormal case is that sonar does not run at all; in this case, heartbeat messages will not arrive at the receiver and the receiver has the opportunity to discover this.)

In the normal error case we have a couple of options, not exclusive:

• ignore it, hope it was transitional, and if not, hope that the failure of heartbeats to arrive at the target will alert somebody
• log to syslog, hope somebody sees it
• log on an error side channel, so that the target can surface the problem when errors arrive

In the interest of making errors actionable I think the third option is best, and it doesn't cost us much.

We are already multiplexing two data streams on the sonar ps stream - the sample stream and the per-system load stream, with the "load" and "gpuinfo" fields - so there's no sin in adding a third, or if you like, a third field to the existing side channel. In the CSV data this would likely be an "error" field that shows up in the first record, along with "load" and "gpuinfo" except it would also show up in heartbeat records, and in JSON data it would just be an additional "error" field at the top level. The field would be absent if there are no errors to report. The payload would be a string. A complication is that we don't want to push literal newlines in either type of data, so the string should be url-encoded or base64-encoded or in some other way safe for transmission. (Neither encoding uses " or , and both are good for both CSV and JSON. Base64 adds about 33% overhead for all data. URL-encoding adds little overhead for ASCII text, which is the expected case.)

Also see #201 #232.

[lars-t-hansen]

I guess we have to consider not just the ps command, even though that is most important, but also the sysinfo and slurm commands. For sysinfo, we're sending one JSON object (or one line of CSV) that can take a new error field. For slurm, we're currently sending multiple lines of CSV or multiple lines of JSON (once #228 lands), but here we'd probably be better off sending a single JSON object with an embedded array of job data, so that fields like the error field can be attached to the enclosing object.

[lars-t-hansen]

We could also use this as an informational conduit, so in addition to an "error" key in the output data there could be an "info" key. This comes up in the case of the ps command, where there are informational messages (lockfile could not be deleted) and some possibly soft errors that should not cause the command to fail.