Skip to content

Commit

Permalink
mptcp.dev v2
Browse files Browse the repository at this point in the history 
Here is a new version of mptcp.dev website for end-users, and apps
developers.

- The home page now also explains what is MPTCP, and the different
  concepts.

- The setup page describes what the users can configure

- The debugging page helps getting more info to check if everything is
  OK.

- The implementation guide is for the apps developers who want to
  support MPTCP natively.

- The MPTCP info page is also for apps devs, to retrieve MPTCP stats
  from the sockets.

- The FAQ page helps answering questions we got in the past, and when
  adding MPTCP in lighttpd and curl.

- The Apps page list apps supporting MPTCP natively.

- The implementation details page is a placeholder for those who want to
  describe how MPTCP is implemented in the Linux kernel.

Thank you to Maxime for the initial draft written with Dorian's help,
and to Olivier for the review.

Co-developed-by: Maxime Dourov (@mux99) <[email protected]>
Co-developed-by: Dorian Craps (@CrapsDorian) <[email protected]>
Reviewed-by: Olivier Bonaventure (@obonaventure) <[email protected]>
Reviewed-by: Mat Martineau (@mjmartineau) <[email protected]>
Signed-off-by: Matthieu Baerts (NGI0) <[email protected]>
  • Loading branch information
@matttbe
matttbe committed Apr 18, 2024
1 parent f6f7406 commit 8bcc012
Show file tree
Hide file tree
Showing 9 changed files with 1,220 additions and 61 deletions.
105 changes: 44 additions & 61 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,61 +1,44 @@
---
layout: home
title: MPTCP
nav_order: 1
---

# Multipath TCP for Linux

Multipath TCP (MPTCP) builds on top of TCP to improve connection
redundancy and performance by sharing a network data stream across
multiple underlying TCP sessions. The MPTCP v1 protocol is defined
in [RFC 8684](https://www.rfc-editor.org/rfc/rfc8684.html).

The Linux MPTCP community develops and maintains the MPTCP v1 stack in
the Linux kernel (v5.6 or later) and associated userspace tools and
libraries.

This site is new and still evolving, so please refer to the [Linux MPTCP Upstream Project wiki](https://github.com/multipath-tcp/mptcp_net-next/wiki) for additional information.

_For out-of-tree kernels before v5.6 and an implementation of the experimental [MPTCP v0](https://www.rfc-editor.org/rfc/rfc6824.html) protocol, see https://multipath-tcp.org/_

## Features

As of Linux v5.19, major features of MPTCP include:

* Support of the `IPPROTO_MPTCP` protocol in `socket()` system calls.
* Fallback from MPTCP to TCP if the peer or a middlebox do not support MPTCP.
* Path management using either an in-kernel or userspace path manager.
* Socket options that are commonly used with TCP sockets.
* Debug features including MIB counters, diag support (used by the `ss` command), and tracepoints.

See the
[ChangeLog](https://github.com/multipath-tcp/mptcp_net-next/wiki/#changelog)
for more details.

## Communication

* Mailing List: [email protected] (and [archives](https://lore.kernel.org/mptcp))
* IRC: [#mptcp](https://web.libera.chat/?nick=mptcp-dev-guest?#mptcp) on libera.chat
* Online [Meetings](https://github.com/multipath-tcp/mptcp_net-next/wiki/Meetings)

## Projects

* Maintained by MPTCP community members
* Kernel development on GitHub: https://github.com/multipath-tcp/mptcp_net-next/
* Multipath TCP Daemon: https://github.com/intel/mptcpd
* The `mptcpd` daemon can do full userspace path management or control the in-kernel path manager.
* Includes the `mptcpize` utility to allow legacy TCP binaries to use MPTCP.
* Packetdrill with MPTCP enhancements: https://github.com/multipath-tcp/packetdrill
* Projects with MPTCP-related enhancements
* [iproute2](https://wiki.linuxfoundation.org/networking/iproute2) (for the `ip mptcp` command)
* [Network Manager](https://networkmanager.dev): MPTCP features are included starting with v1.40.
* [Multipath TCP applications](https://github.com/mptcp-apps/): A project to coordinate MPTCP updates for popular TCP applications.

## Kernel Development

* [Git Repository](https://github.com/multipath-tcp/mptcp_net-next.git) ([branch descriptions](https://github.com/multipath-tcp/mptcp_net-next/wiki/Git-Branches))
* [Issue tracker](https://github.com/multipath-tcp/mptcp_net-next/issues)
* [Patchwork](https://patchwork.kernel.org/project/mptcp/)
* [Continuous Integration](https://github.com/multipath-tcp/mptcp_net-next/wiki/CI)
* [Testing](https://github.com/multipath-tcp/mptcp_net-next/wiki/Testing)
# mptcp.dev website

This README is meant as a guide on the structure and particularities of this
website for those who want to contribute.

## The pages
- `index`, the landing page should contain basic context of this website and
explanations on what MPTCP is. Also a section dedicated to links to other
MPTCP related content.
- `setup` is meant as a guide for end users, helping them setup and use MPTCP on
their system.
- `debugging` is meant as a source of solutions on how and what to use to debug
MPTCP on a system.
- `implementation` contains information on how to support MPTCP natively in an
app on Linux.
- `mptcp-info` contains info on how app devs can retrieve MPTCP info.
- `faq`, for Frequently Asked Questions.
- `apps`, the list of currently apps supporting MPTCP natively.
- `details` contains detailed explanations of the kernel implementation.

## Front matter
At the top of all pages a section between two `---` lines contains information
on the page.

All `jekyll` and `just the docs` features are supported, and new ones have been
added:
- `nav_titles` is a boolean. When set to true, the markdown titles of the page
will be displayed in the navbar for easy access to a specific section.
- `titles_max_depth` is an integer that indicates the maximum level of titles
included in the navbar. (*it directly corresponds to the number of* `#`)

## Liquid markers
- `{: .ctsm}`, can be used on `<details>` tags to display "`(click to see more)`"
in grey text after the `summary`

- `{: warning}`, `{: note}`, `{: info}` are callouts. Used before or after a
paragraph, they will highlight it.

## Mermaid
`mermaid` graphs are supported in markdown files.

## Contributions
This website is open to contributions via pull request. A link at the bottom
left of each page allows anyone to edit it on GitHub.
48 changes: 48 additions & 0 deletions apps.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
layout: home
title: Apps supporting MPTCP
nav_order: 7
nav_titles: true
titles_max_depth: 2
---

Applications listed bellow support MPTCP natively. Please note that other apps
can be forced to use it by following [these instructions](setup.html#force-applications-to-use-mptcp).

## Linux apps

| Name | Version | How to use |
| --- | --- | --- |
| [Lighttpd 1.4](https://www.lighttpd.net/) | [1.4.76+](https://github.com/lighttpd/lighttpd1.4/pull/132) | [`server.feature-flags = ( "server.network-mptcp" => "enable" )`](https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_feature-flagsDetails) |
| [Apache Traffic Server](https://trafficserver.apache.org/) | [9.2.4+](https://github.com/apache/trafficserver/pull/10701) | [`server_ports: <port> mptcp`](https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.yaml.en.html) |
| [Envoy](https://www.envoyproxy.io/) | [1.21.0+](https://github.com/envoyproxy/envoy/pull/18780) | [`"enable_mptcp": "true"`](https://www.envoyproxy.io/docs/envoy/v1.21.6/api-v3/config/listener/v3/listener.proto#envoy-v3-api-field-config-listener-v3-listener-enable-mptcp) (`listening` config) |
| [QEmu](https://www.qemu.org/) | [6.1+](https://lore.kernel.org/qemu-devel/[email protected]/) | [`<ip>:<port>,mptcp`](https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-48) |
| [Shadowsocks libev](https://github.com/shadowsocks/shadowsocks-libev) | [v3.3.6+](https://github.com/shadowsocks/shadowsocks-libev/pull/2902) | [`--mptcp`](https://github.com/shadowsocks/shadowsocks-libev) |
| [Shadowsocks Rust](https://github.com/shadowsocks/shadowsocks-rust) | [v1.16.0+](https://github.com/shadowsocks/shadowsocks-rust/pull/1157) | [`--mptcp`](https://github.com/shadowsocks/shadowsocks-rust) |

## Tools

| Name | Version | Comment |
| --- | --- | --- |
| [TCPDump](https://www.tcpdump.org/) | [4.5.0+](https://github.com/the-tcpdump-group/tcpdump/commit/578dd316f3) | Can decode MPTCP options in TCP packets |
| [Wireshark](https://www.wireshark.org/) | [4.2.4+](https://github.com/wireshark/wireshark/commit/3bc42dbf8e) | Can decode and filter MPTCP options |
| [Syzkaller](https://github.com/google/syzkaller) | [Feb 2020](https://github.com/google/syzkaller/pull/1579) | Can exercise MPTCP sockets and Netlink APIs |

## MacOS apps

| Name | Version | How to use |
| --- | --- | --- |
| [iperf3-darwin](https://software.es.net/iperf/) | [11.0+](https://developer.apple.com/documentation/foundation/nsurlsessionmultipathservicetype?language=objc) | `--apple-multipathsvc` |
| SSH | [OpenSSH-281.40.2](https://github.com/apple-oss-distributions/OpenSSH) | `applemultipath` option |
| [Shellfish](https://secureshellfish.app) | ? | Settings / Advanced / 🌍 Multipath TCP |

## Add native MPTCP support

Do not hesitate to add native MPTCP support in more apps! For some ideas, feel
free to look at this [spreadsheet](https://docs.google.com/spreadsheets/d/1F2-v4Dhdn0rMyJZ3m5chyNiwg7oj0rpSR11GEykatJw/edit#gid=0).

## Others?

Please open a new [pull request](https://github.com/multipath-tcp/mptcp.dev/pulls)
or an [issue](https://github.com/multipath-tcp/mptcp.dev/issues) if you know any
other apps with a community of users and supporting MPTCP natively.
63 changes: 63 additions & 0 deletions debugging.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
layout: home
title: Debugging
nav_order: 3
nav_titles: true
titles_max_depth: 2
---

To verify that everything is working correctly, or to debug issues, multiple
tools can be used.


## ss

The [`ss`](https://www.man7.org/linux/man-pages/man8/ss.8.html) command line
tool on Linux systems has an option to list MPTCP sockets: `-M`. The recommended
options are: `ss -Mani`.

- `M`, MPTCP sockets
- `a`, display all socket not just "established" ones
- `n`, prevent the port number to protocol conversion
- `i`, connection info matching that from Netlink Diag interface: `TCP_INFO`,
`MPTCP_INFO`, etc.

{: .note}
Note that technically, each subflow (path) is a TCP connection. To see subflows,
`ss -tani` can be executed as root: subflows will be marked with `tcp-ulp-mptcp`.

`ss` can be used with extra filters, e.g. to restrict by connection states, IP
addresses, ports, devices, marks, CGroup, etc.


## ip mptcp

[`ip mptcp`](https://man7.org/linux/man-pages/man8/ip-mptcp.8.html) can be used
to configure the MPTCP path-manager, but also to monitor path-manager events:

```bash
ip mptcp monitor
```


## nstat

[`nstat`](https://www.man7.org/linux/man-pages/man8/nstat.8.html) can list
kernel MIB counters. MPTCP has a bunch of them, e.g.

```bash
nstat -asz | grep MPTcpExt
```

For more details about these counters, see:
[MPTCP Doc](https://mptcp-apps.github.io/mptcp-doc/mptcp-linux.html#analyzing-the-output-of-nstat).

When encountering an issue with MPTCP, it is often interesting to run `nstat -n`
before, and `nstat | grep Tcp` after the issue.


## TCPDump and WireShark

[TCPDump](https://www.tcpdump.org) and [WireShark](https://www.wireshark.org)
support MPTCP: they can be use to take packet traces, and recent enough versions
will show MPTCP options in TCP packets if present.
22 changes: 22 additions & 0 deletions details.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
layout: home
title: Implementation details (Kernel)
nav_order: 8
nav_titles: false
titles_max_depth: 2
---

The MPTCP protocol is described in [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684),
and also in the [MPTCP Doc](https://mptcp-apps.github.io/mptcp-doc/mptcp.html),
but neither of those describe the internals of the MPTCP implementation in the
Linux kernel.

A new socket type has been added for MPTCP for the userspace-facing socket.
The kernel is in charge of creating subflow sockets: they are TCP sockets
where the behavior is modified using TCP-ULP.

MPTCP listen sockets will create "plain" *accepted* TCP sockets if the
connection request from the client didn't ask for MPTCP.

This page needs to be improved, feel free to
[contribute](https://github.com/multipath-tcp/mptcp.dev/edit/main/details.md).
155 changes: 155 additions & 0 deletions faq.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
layout: home
title: FAQ
nav_order: 6
nav_titles: true
titles_max_depth: 2
---

## Are there any security & privacy concerns?
MPTCP aims to maintain the same level of security as traditional TCP, with
specific mechanisms to counter common network attacks. Find out more in
[RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684#name-security-considerations).
To be more secure than TCP, some modifications of the protocol would be needed,
e.g. [MPTCPsec](https://inl.info.ucl.ac.be/system/files/infocom_mptpcsec.pdf).

## Why & when should MPTCP be enabled by default?
<details markdown="block">
<summary>Here, servers and clients must be considered separately: MPTCP can be
enabled by default on servers, to be used only when requested, with a minimal
impact. On the client side, it might be useful to notify users that MPTCP is
being used by default. </summary>

- Clients typically have the most to gain by using MPTCP, but the benefits of
MPTCP are mainly realized when users have
[configured](setup.html#using-multiple-ip-addresses) their system to make use
of its multipath capability. Still, even when only one network interface is
available, MPTCP can be helpful in mobility use-cases that involve frequent
switching from one network to another without stopping the connections. When
servers don't support MPTCP, the connection continues in "plain" TCP.

- Servers usually don't directly benefit from MPTCP, due to their stable, fast,
and reliable network connections. The client/server system as a whole is where
the typical MPTCP improvements are experienced. There are server-specific use
cases as well:
- Switching from one network to another without a disconnection that would
restart a long operation
- Faster throughput by aggregating multiple TCP flows
- Lowering latency by sending data in parallel on multiple subflows, so the
lowest-latency path "wins". (Note: The MPTCP protocol allows for this but
the Linux packet scheduler does not yet implement such a feature)
- etc.
We recommend
enabling MPTCP on servers by default to let users choose whether to use
MPTCP. When clients don't request to use MPTCP, server applications will
create "plain" TCP sockets within the kernel when connections are `accept`ed,
making the performance impact minimal.
</details> {: .ctsm}

## Are there any performance impacts when using MPTCP?
MPTCP is engineered to improve network resilience and utilization without
adversely affecting the performance of TCP applications. It adds a few bytes in
each TCP packet, causing a manageable overhead (~1%) that becomes advantageous
when leveraging multiple network paths, potentially increasing throughput and
reliability.

It is also important to note that, when clients don't request to use MPTCP,
server applications will create "plain" TCP sockets within the kernel when
connections are `accept`ed, making the performance impact minimal.

## Are there unsupported TCP socket options?
MPTCP supports most TCP socket options. It is possible some less-common ones are
not supported. If it is the case, please document your use-case in a new
[issue](https://github.com/multipath-tcp/mptcp_net-next/issues/).

For example, MPTCP in the Linux kernel is
[currently](https://github.com/multipath-tcp/mptcp_net-next/issues/480) not
compatible with KTLS.

## What are the supported operating systems?
<details markdown="block">
<summary>MPTCP is supported in the official Linux kernel starting with version
5.6. Any applications can [easily use it](implementation.html). The adoption of
MPTCP extends beyond to various platforms including macOS. But... </summary>

The use of MPTCP on macOS differs from Linux:
- It is most straightforward when applications use the system's
[frameworks](https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/improving_network_reliability_using_multipath_tcp)
- There is some documentation for
[connectx](https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/man/man2/connectx.2.auto.html)
and
[disconnectx](https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/man/man2/disconnectx.2.auto.html)
system calls that mention multipath support, however it is not clear what is
required to use these interfaces. There does not appear to be public example
code.

On FreeBSD, there was an ongoing implementation, but that was years ago, and it
is not working today according to
[this](http://www-cs-students.stanford.edu/~sjac/freebsd_mptcp_info.html).

There are other implementations, but on specific systems (Citrix load balancer,
userspace, etc.): more details
[here](http://blog.multipath-tcp.org/blog/html/2018/12/15/apple_and_multipath_tcp.html).

It is possible to use MPTCP on Windows with
[WSL2](https://perso.uclouvain.be/tom.barbette/mptcp-on-windows-with-wsl2/).
</details> {: .ctsm}

## MPTCP vs. QUIC
MPTCP enhances TCP's functionality at the transport layer by enabling multipath
capabilities, whereas QUIC, built atop UDP, focuses on reducing latency and
improving connection migration. While both propose multipath functionality,
their development and standardization stages differ.

Multipath capability in QUIC is not yet standardized. Here is the
[draft](https://quicwg.org/multipath/draft-ietf-quic-multipath.html).
Applications might have more configuration options to be able to select
different paths, and control data flow over those paths.

## MPTCPv0 vs. MPTCPv1
There are two different versions of the protocol:
[RFC 6824](https://datatracker.ietf.org/doc/html/rfc6824) (MPTCPv0, obsolete) and
[RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) (MPTCPv1, current).
The upstream Linux kernel only supports v1. A previous
[out-of-tree kernel](https://github.com/multipath-tcp/mptcp) supported both
v0 and v1, but it is now recommended to use the
[upstream](https://github.com/multipath-tcp/mptcp_net-next/) releases instead.

## What about middleboxes?
MPTCP is meticulously designed to ensure fallback to standard TCP when necessary.
This ensures uninterrupted connectivity amidst the presence of NATs and
firewalls (middleboxes) that might remove "unknown" TCP options like MPTCP.

If you notice that MPTCP is not allowed in your network, and "plain" TCP is used
instead, please report this issue to the people in charge of your network: it is
very likely a mistake that MPTCP is not allowed.

## How should applications handle missing MPTCP support?
Applications supporting MPTCP natively should first try to create an MPTCP
socket, then fallback to "plain" TCP in case of errors. If the Linux kernel does
not support MPTCP, a proper error will be returned when creating the socket. The
kernel version and its config can be different at build-time and at run-time
(e.g. some builders are using chroot/containers with up-to-date software
programs, but an old stable kernel). So it is recommended to do such checks at
run-time, letting the kernel returning an error if it is not supported, rather
than trying to guess at build-time what the end-user will have at run-time.

## What build time checks are needed/recommended?
Since a common practice is to compile source code on a different machine,
potentially with an older kernel version running on the build machine, it is
recommended not to restrict MPTCP utilization at build-time other than checking
than the target is Linux.

Note that it might be necessary to manually define `IPPROTO_MPTCP`, because old
libC versions might not have it. See the [Implementation Guide](implementation.html)
for more details about that.

## <code>IPPROTO_MPTCP</code> is not defined
Since a program is not always compiled on the system it runs on, it is
recommended to manually define `IPPROTO_MPTCP` if the symbol is not already
defined:
```js
#ifndef IPPROTO_MPTCP
#define IPPROTO_MPTCP 262
#endif
```
Loading

0 comments on commit 8bcc012

Please sign in to comment.