Messages can be received out of order across multiple channels

[brt-colleen]

I have a process receiving messages from multiple processes (messages are published on different channels). I have a handler waking up for each message as I receive it. If I look at monotonic time in each handler, it does not increase monotonically.

In lockless_queue there's a race between publishing and making the data visible.

[philsc]

Would it be fair to rename the ticket to be "can be received out of order"?

[AustinSchuh]

Yea. It is unclear exactly when in the process a message is "sent". The key here is multiple channels, and what the receiver observes.

[AustinSchuh]

Thinking out loud, we should be able to make time-stamping happen after the compare and exchange which does publishing.

The key here is any subsequent operation (reading or writing) needs to fill in the timestamps before proceeding. The original sender needs to also attempt to timestamp. The message needs to have a reserved value in it when originally published. The first writer of the timestamp wins, and everyone else gives up. They all need to do them in order too, monotonic then rt. The truck is to make sure performance doesn't suffer too much, both average and peak.

We also need to make sure all fetchers and watchers treat the same next state as available. That can be the publish compare and exchange, or updating the next pointer. Otherwise we could go backwards in even more subtle ways.

Today, we provide no guarantees about how much delay can be between when the rt and monotonic clocks get sampled. The RT is guidance. If we do the writes as 2 64 bit atomics writes, we arent making it worse.

The roborio is the only 32 bit platform around, and is going eol soon. @jkuszmaul and I vote that we don't fix this on 32 bit machines.

The real challenge is testing. The death test infrastructure should serve us well, along with queue racer.