Redesign MQTT examples

[petervdonovan]

The following is copied from a thread here. See the linked PR for comments from Edward and Erling.

The MQTT API is not thread-safe. If this is the case, then why not enforce mutual exclusion of accesses to the MQTT API by requiring all accesses to the MQTT API to be performed within the same reactor instance?

In order for this approach to work, it would be necessary for reactors to pass various arguments along to the MQTT reactor, and the MQTT reactor would have to pass back any handles and outputs that it gets from the API, and everything would have to be wired to that one reactor, and altogether it would make for a cumbersome way to access a simple C API. Furthermore, we would need some way of specifying that a reactor can only be instantiated once, which will be a nontrivial problem to solve at compile time in the presence of mutations or recursive instantiation with modal models.

However, I think that we should consider this approach anyway because doing so would draw attention to the areas where the LF language could be improved. If the standard way of using LF is to use it to structure an application but to resort to explicit critical sections as soon as it becomes inconvenient to use LF to avoid concurrency bugs, then LF applications will not be safer than applications written in popular general-purpose languages.

[edwardalee]

One issue is that MQTTPublisher and MQTTSubscriber could not be separate reactors. This will create programs with more zero-delay loops (any time you want to react to a subscription message by publishing something), which will make these programs incompatible with enclaves and will limit their parallelism.