4.1.2 Flood Test
This test was made to compare the Arrowhead Framework with current communication frameworks in distributed systems, on how they handle clouds with a large number of similar service providers.
The two other solutions we chose for this were MQTT and DDS. The representatives for these technologies for this test are:
- For MQTT: the Eclipse Mosquitto server with clients written using Eclipse Paho
- For DDS: RTI DDS Connext
- And for Arrowhead: The server reference implementation and client frameworks
The test setup is simple:
A consumer want some metric. This metric is not available directly, but it can be calculated from the data of two different kinds of sensors. However, this consumer is not able to make this calculation itself, therefore it requests a processor service. Unfortunately, everyone and their mother provides this service. For convenience, the client which asks for the processing service is always the Consumer, the ones offering it are the Processors, and the sensors are simply named Sensor A and Sensor B. This is regardless of what specific role or roles they have in their respective frameworks' terminologies.
The Consumer sends the request, the Processor receives it, and sends requests to the sensors, and they send it their data. When it's done with the calculations, it responds to the Consumer. The consumer timed the exchange from first request to, and first response from the Processor. Also, every time a system sends or receives a message, it publishes this information. From these, we can tell how much effort went into this simple task.
The components run on three machines.
Whenever a tested component send or receives a message, it publishes this information using MQTT. Only the Auxiliary is subscribed to these topics. The Processor Factory creates and starts a processor whenever the Test Organizer sends a message to it (using MQTT).
The test is run separately on each framework. Firstly, if applicable, we start the Mosquitto server and the Arrowhead Core systems. Then we start the Test Organizer, Consumer, Sensors A and B, and the Processor Factory.
Then, the cycle begins:
The Test Organizer creates the Auxiliary, and sends a message to the Processor Factory to create a new Processor. Then the consumer sends the first request and makes a timestamp. The messages are fired as needed, and when the first successful processor responds to the consumer, another timestamp is made. Then wait a little for everything to settle down.
The output is then written in a .csv file with name corresponding to the framework tested. It consists of the number of the cycle (how many processors), the beginning and end timestamps (ISO-8601), time (milliseconds), number of messages sent, and received.
Then this cycle is repeated to taste.
Machine A was
Machine B was
Machine C was