doc: update shm_zerocopy.rst

doc/rst/advanced/layers/shm_zerocopy.rst

@@ -8,12 +8,12 @@ eCAL Zero Copy
 
 .. note:: 
 
-   The eCAL Zero Copy mode has been added in:
+   The eCAL Zero Copy mode was introduced in two steps:
 
-   - eCAL 5.10 (Zero-copy **subscriptions**)
-   - eCAL 5.12 (Zero-copy **publishing**)
+   - eCAL 5.10 (zero-copy on subscriber side only, still one memory copy on the publisher side)
+   - eCAL 5.12 (zero-copy on publisher and subscriber side, see Full Zero Copy Behavior)
 
-   It is turned off by default.
+   In all versions it is turned off by default.
 
 Enabling eCAL Zero Copy
 =======================
@@ -29,7 +29,7 @@ Enabling eCAL Zero Copy
 
 - **Use zero-copy for a single publisher (from your code):**
 
-  Zero-copy can be activated for a single publisher from the eCAL API:
+  Zero-copy can be activated (or deactivated) for a single publisher from the eCAL API:
 
   .. code-block:: cpp
 
@@ -46,12 +46,11 @@ Enabling eCAL Zero Copy
 
   If you want to avoid this copy, you can use the :ref:`low-level API <transport_layer_shm_zerocopy_low_level>` to directly operate on the SHM buffer.
 
-Zero Copy behavior
+Full Zero Copy behavior
 ==================
 
-The bevior for publishers differs depending on the used transport layers.
-When publishing data over network, those connections cannot make use of any Zero copy features.
-
+The eCAL Zero Copy mechanism is working for local (inner-host) publish-subscribe connections only. Sending data over a network connection will no 
+Sending data over a network cannot benefit from that feature.
 
 Shared-Memory-only connection
 -----------------------------
@@ -60,29 +59,31 @@ This describes the case, where a publisher publishes it's data **only via shared
 
 **Publisher**:
 
-- Protobuf:
+- Protobuf API Level:
 
   #. The user sets the data in the **protobuf object**
 
   #. The publisher **locks** the SHM buffer.
 
      *This operation may take some time, as the publisher needs to wait for the subscriber to release the buffer.*
-     *This can be relaxed by using th multi-buffering feature.*
+     *This can be relaxed by using the multi-buffering feature.*
 
   #. The publisher **serializes** the protobuf object **directly into the SHM** buffer
 
      *Due to the technical implementation of protobuf, this will cause the entire message to be serialized and re-written*
 
   #. The publisher **unlocks** the SHM buffer
 
-- Low Level Memory access:
+- Binary API Level:
 
   #. The publisher **locks** the SHM buffer
 
   #. The user **directly** writes data to the **SHM buffer**.
 
   #. The publisher **unlocks** the SHM buffer
 
+  #. The publisher **informs** all connected subscriber
+
 **Subscriber**:
 
 #. The subscriber **locks** the SHM buffer
@@ -97,22 +98,39 @@ This describes the case, where a publisher publishes it's data **only via shared
 Mixed Layer connection
 ----------------------
 
-This describes the case where a publisher publishes it's data via **shared memory and network**.
+This describes the case where a publisher publishes its data parallel via **shared memory and network** (tcp or udp). So we have at least one local subscription and one external (network) subscription on the provided topic.
 
 **Publisher**:
 
-The publisher cannot work in SHM Zero Copy mode, if there are non-SHM connections.
-In this case, the publisher will fall back to the **normal SHM** mode.
+Regardless of whether the data is generated by a Low Level Binary Publisher or by a Protobuf Publisher, it is always written to an process internal cache first. This memory cache is then passed sequentially to the connected transport layers "shared memory", "innerprocess", "udp" and "tcp" in this order.
+
+Compared to the Full Zero Copy behavior described above with only local (shm) connections, we have a copy of the user payload on the publisher side again.
+
+This leads to the following publication sequence for a local connection:
 
-#. For Protobuf: The user sets the data in the **protobuf object**
+- Protobuf API Level:
+
+  #. The user sets the data in the **protobuf object**
 
-#. For Low Level Memory access: The user modifies parts of the SHM buffer. 
+  #. The publisher **serializes** the protobuf object into a process internal data cache
+
+  #. The publisher **locks** the SHM buffer.
+
+  #. The publisher **copies** the process internal data cache to the SHM buffer.
+
+  #. The publisher **unlocks** the SHM buffer
+
+- Binary API Level:
+
+  #. The publisher **copies** the binary user data into a process internal data cache
+
+  #. The publisher **locks** the SHM buffer
 
-#. The publisher **locks** the SHM buffer
+  #. The publisher **copies** the process internal data cache to the SHM buffer.
 
-#. The publisher **copies** the private SHM buffer or the serialized protobuf object to all data layers, one of which is the SHM buffer.
+  #. The publisher **unlocks** the SHM buffer
 
-#. The publisher **unlocks** the SHM buffer
+  #. The publisher **informs** all connected subscriber
 
 **Subscriber**:
 
@@ -130,56 +148,134 @@ So they will **directly** read from the SHM buffer.
 Low Level Memory Access
 =======================
 
-For unleashing the full power of eCAL Zero Copy, the user needs to directly work on the eCAL Shared Memory via the ``CPayload`` API.
+For unleashing the full power of Full eCAL Zero Copy, the user needs to directly work on the eCAL Shared Memory via the ``CPayloadWriter`` API. The idea behind the new ``CPayloadWriter`` API is to give the user the possibility to modify only the data in the memory that has changed since the last time the date was sent. The aim is to avoid writing the complete memory and thus save computing time and reduce the latency of data transmission.
+
+The new payload type ``CPayloadWriter`` looks like this (all functions unnecessary for the explanation have been omitted):
+
+.. code-block:: cpp
+
+  class CPayloadWriter
+  {
+  public:
+    // the provisioned memory is uninitialized ->
+    // perform a full write operation
+    virtual bool Write(void* buffer_, size_t size_) = 0;
+
+    // the provisioned memory is initialized and contains the data from the last write operation ->
+    // perform a partial write operation or just modify a few bytes here
+    //
+    // by default this operation will just call `Write`
+    virtual bool Update(void* buffer_, size_t size_) { return Write(buffer_, size_); };
+
+    // provide the size of the required memory (eCAL needs to allocate for you).
+    virtual size_t GetSize() = 0;
+  };
+
+The user must derive his own playload data class and implement at least the ``Write`` function. This ``Write`` function will be called by the low level eCAL SHM layer when finally the connected memory file needs to be written the first time (initial full write action). 
+
+For writing partial content (modifying the memory content) the user has to define a second function called ``Update``. This function is called by the eCAL SHM layer if the connected memory file is in an initialized state i.e. if it was written with the previously mentioned ``Write`` method. As you can see, the ``Update`` function simply calls the ``Write`` function by default if it is not overwritten.
 
-**Normal Serialization**
+The implementation of the ``GetSize`` method is mandatory. This method is used by the eCAL SHM layer to obtain the size of the memory file that needs to be allocated.
 
-- Overwrites entire memory => kind of a 1-copy instead of 2-copy approach.
+**Example**:
 
-- Easy to use, ships with eCAL
+The following primitive example shows the usage of the ``CPayloadWriter`` API to send a simple binary struct efficient by implementing a full ``Write`` and an ``Update`` method that is modifying a few struct elements without memcopying the whole structure again into memory.
 
-- This is meant for message serializations like protobuf.
-  Those can then directly serialize into the SHM buffer.
+This is the customized new payload writer class. The ``Write`` method is creating a ``SSimpleStruct`` struct and will copy the whole structure into the opened shared memory file buffer. The ``Update`` method gets a view of the opened shared memory file, and applies modifications on the struct elements ``clock`` and ``bytes``.
 
-**Partial Serialize**
+.. code-block:: cpp
 
-- Despite the name, this this the **Direct memory access** method, that is meant for **non-serialization** message formats
+  #pragma pack(push, 1)
+  struct SSimpleStruct
+  {
+    const uint32_t version = 7;
+    const uint16_t rows    = 5;
+    const uint16_t cols    = 3;
+    uint16_t clock         = 0;
+    uint8_t  bytes[5 * 3]  = { 0 };
+  };
+  #pragma pack(pop)
 
-- Has possibility to only change parts of the memory
+  class CStructPayload : public eCAL::CPayloadWriter
+  {
+  public:
+    bool Write(void* buf_, size_t len_) override
+    {
+      // write complete content to the shared memory file
+      if (len_ < GetSize()) return false;
 
-- Always operates on 2 Buffers that **needs to be kept in sync**
+      // write the complete struct into memory
+      SSimpleStruct simple_struct;
+      memcpy(buf_, &simple_struct, GetSize());
 
-  - There is a "private" buffer in the CPayload that is modified
+      return true;
+    };
 
-  - There is a "public" SHM buffer that the user needs to **manually** keep in sync with the private buffer via partial serialize
+    bool Update(void* buf_, size_t len_) override
+    {
+      // update content of the shared memory file
+      if (len_ < GetSize()) return false;
 
-  - **The user needs to make sure that both buffers are kept in sync. eCAL does not offer any help with that. A call to PartialSerialize always needs to make sure that the public buffer is updated to the private buffer.**
+      // cast existing memory to a SSimpleStruct
+      SSimpleStruct* simple_struct = static_cast<SSimpleStruct*>(buf_);
 
-- Example:
+      // modify the simple struct in memory
+      simple_struct->clock++;
+      for (auto i = 0; i < (simple_struct->rows * simple_struct->cols); ++i)
+      {
+        simple_struct->bytes[i] = static_cast<char>(simple_struct->clock);
+      }
 
-  - User writes 1 byte of the message (-> is written in private buffer)
+      return true;
+    };
 
-  - User writes 1 other byte of the message (-> also written in private buffer)
+    size_t GetSize() override { return sizeof(SSimpleStruct); };
+  };
 
-  - User calls ``Send()``
+To send this payload you just need a few lines of code:
 
-  - eCAL calls the ``PartialSerialize()`` function that the User needs to implementat
+.. code-block:: cpp
 
-  - The ``PartialSerialize()`` now needs to know which bytes have been changed prior to to the ``Send()`` call and also change those in the public buffer
+  int main(int argc, char** argv)
+  {
+    // initialize eCAL API
+    eCAL::Initialize(argc, argv, "binary_zero_copy_snd");
 
-Zero Copy vs. normal eCAL SHM
+    // publisher for topic "number"
+    eCAL::CPublisher pub("simple_struct");
+
+    // turn zero copy mode on
+    pub.ShmEnableZeroCopy(true);
+
+    // create the simple struct payload
+    CStructPayload struct_payload;
+
+    // send updates every 100 ms
+    while (eCAL::Ok())
+    {
+      pub.Send(struct_payload);
+      eCAL::Process::SleepMS(100);
+    }
+
+    // finalize eCAL API
+    eCAL::Finalize();
+
+    return(0);
+  }
+
+Default eCAL SHM vs. Full Zero Copy SHM
 =============================
 
-.. list-table:: Zero Copy vs. normal eCAL SHM
+.. list-table:: Default eCAL SHM vs. Full Zero Copy SHM
    :widths: 10 45 45
    :header-rows: 1
    :stub-columns: 1
 
    * -
 
-     - eCAL SHM - Default
+     - Default eCAL SHM
 
-     - eCAL SHM - Zero Copy
+     - Full Zero Copy SHM
 
    * - Memcopies
 
@@ -215,4 +311,6 @@ Zero Copy vs. normal eCAL SHM
        - Publishers need to keep the the SHM buffer locked while performing the message serialization
 
 Combining Zero Copy and Multibuffering
-======================================
+======================================
+
+For technical reasons the Full Zero Copy mode described above is turned of if the Multibuffering option ``CPublisher::ShmSetBufferCount`` is activated. To support differential/partial writes on a set of connected memory files can not be implemented in an efficient way. Default (subscriber side) Zero Copy is working in combination with Multibuffering as described.