SILKIT-1608: disable nagle algorithm by default

[VJonasHolley]

Subject

• Disable nagle algorithm by default (prevent bad interaction with delayed acks)
• Add performance section to troubleshooting in docs

Description

See: https://jira.vg.vector.int/browse/SILKIT-1608

[VDanielEdwards]

Let's start a discussion! 🗣️

[VDanielEdwards]

Suggested change

	Configuration of TCP Stack (Kernel)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	TCP Socket Configuration
	~~~~~~~~~~~~~~~~~~~~~~~~

[VDanielEdwards]

Suggested change

	The TCP stack of the kernel can be configured via the participant configuration file, more precisely the :doc:`middleware<../configuration/middleware-configuration>` options.
	The TCP sockets used by SIL Kit can be configured via the participant configuration file, more precisely the :doc:`middleware<../configuration/middleware-configuration>` options.

[VDanielEdwards]

Suggested change

	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	~~~~~~~~~~~~~~~~~~~

[VDanielEdwards]

Suggested change

	This configuration needs to be done carefully. In particular, Nagle's algorithm (``TcpNoDelay``) and delayed acknowledgements (``TcpQuickAck``) are known to interact badly.
	This configuration needs to be done carefully. In particular, using Nagle's algorithm (``TcpNoDelay: false``) and delayed acknowledgements (``TcpQuickAck: false``) are known to interact badly.

I would add which configuration values interact badly.

It's also only a 'bad' interaction, if latency is a major concern - which is the case for us, but, e.g., not for bulk transfers (downloads). Nagle and delayed ACKs exist to use bandwidth efficiently, i.e., to use the local buffers as much as possible and only transmit when enough data for 'full' packets is queued and enough space is available for reception.

[VJonasHolley]

I agree, adding the values is really helpful!

A short comment on the 'bad' interaction: It is not only a concern when it comes to latency. I observed problematic behaviour in the following situations:

• Latency peaks (40ms): To be reproduced with LatencyDemo (with small message size)
• Large throughput jitter (standard deviation decreases by one order of magnitude, if Nagle is disabled): Reproduce with BenchmarkDemo (100kB, 10 messages/sim. step)
• Stuck communication: Reproduce with BenchmarkDemo (1 byte, 1 msg/sim. step)

[VDanielEdwards]

I think changing the default value is a good idea. However, this will 'surprise' everyone who hasn't explicitly set this configuration value to false. We should carefully evaluate if this could be an issue. We should also think about how this should influence our version number - or the configuration SchemaVersion (my gut reaction is to increment the SchemaVersion, but treat this specifically as a patch for SIL Kit).

[VDanielEdwards]

However, we have no prescedent for incrementing the SchemaVersion like this. And we also need some code to actually handle varying defaults (e.g.) in different SchemaVersions.

[VDanielEdwards]

The main issue I see in changing the default value, is for users that have configurations where they only set TcpQuickAck: true but do not specify TcpNoDelay (because they relied on the old default).

[VJonasHolley]

I agree, users who did not explicitly set the TCP middleware options may experience different behaviour.
However, we only change the default config value, not the code itself. Does that justify a version increment?
Maybe, a short remark in the release notes is already sufficient?

[VDanielEdwards]

The old formulation is correct! TcpNoDelay: true means to enable the TCP_NODELAY flag, which disables Nagle.

auto MakeAsioSocketOptionsFromConfiguration(const SilKit::Config::ParticipantConfiguration& participantConfiguration) -> SilKit::Core::AsioSocketOptions
{
    SilKit::Core::AsioSocketOptions socketOptions{};
    // ...
    socketOptions.tcp.noDelay = participantConfiguration.middleware.tcpNoDelay;
    // ...
}

if (socketOptions.tcp.noDelay)
{
    socket.set_option(asio::ip::tcp::no_delay{true}, errorCode);
    // ...
}

[VJonasHolley]

Yes!

[VDanielEdwards]

Suggested change

	Per default, Nagle's algorithm is disabled and delayed acknowledgements are enabled. Note that enabling both features may cause severe performance losses as well as non-deterministic behaviour.
	Please note that version X.Y.Z and following will use the ``NO_DELAY`` flag (disable Nagle's algorithm) by default (``TcpNoDelay: true``).
	Using ``TcpNoDelay: true`` and ``TcpQuickAck: true`` together, may cause severe performance losses as well as non-deterministic behaviour.

I would probably reword this to get away from using 'enabled' and 'disabled' so much - it at least confused me a fair bit 😹

[VJonasHolley]

Excellent hint! I was getting confused as well 😁