Subutai Network

The SubOS networking configuration is a key component of Subutai Social. It allows containers on peers around the world to connect, and seem as though they're located on the same LAN.

Many different components combined together to make this system work.

##Resource Host Network Each physical or virtual Resource Host (RH) should have a direct connection to the network. It should be connected to Internet to communicate with remote peers.

RHs require a DHCP service on the network. The RH will use it to acquire an IP-Address, the gateway, and DNS-servers.

###VirtualBox Network Resource Hosts can be physical or virtual machines on any kind of hypervisor. We created an easy to use installer for all operating systems which defaults to using VirtualBox. These installers will create the virtual machine with a following network configuration:

• the Bridged interface is the main port for getting the RH connected to the LAN and the Internet.
• NAT and Host-only adapters are reserved to start working in case the Bridged interface can't be used for normal operation.

###Roaming If the system cannot use the Bridged interface, fails to get an address from the DHCP-server or has some other problem, it will automatically reconfigure the network to start working over the VirtualBox NAT network as a reserve.

The resource host constantly checks network availability and changes network configuration to work with best options available at that moment. It allows us to migrate the RH between networks when users roam and automatically restoring normal operations. This is important if you create a peer on a laptop for example.

###Resource Host Networks Resource Host networks are organized in the following way:

• wan - The network interface used for primary access to the LAN, Internet and other peers and RHs. All traffic from the RH or container goes through this interface.
• mng-net - The network interface used for isolated management traffic between Resource Hosts, and the management server in the same peer.
• nat - The network interface used for providing internet access via NAT to offline containers, which are not connected to any environment.

##Open vSwitch Open vSwitch (OVS) is installed on each Resource Host. It allows us to split the network into separate virtual bridges with its own uplinks and ports.

A fresh newly installed peer without any environment on it yet has following OVS configuration:

Bridge wan
    Port mng-net
        tag: 2
        Interface mng-net
            type: internal
    Port wan
        Interface wan
            type: internal
    Port "eth0"
        Interface "eth0"
Bridge lxc-br
    Port nat
        Interface nat
            type: internal
    Port lxc-br
        Interface lxc-br
            type: internal

##VLAN VLAN is used as a network isolation mechanism in Subutai's SubOS to prevent containers across independent environments from interacting with each other.

• One VLAN is used for management traffic;
• Other VLANs are used to isolate environments from one another;
• Untagged traffic is used for accessing Internet. Additionally it is used for offline containers NOT belonging to any environment.

Containers located in a single VLAN represent an environment. Communication between different environments are NOT allowed to implement isolation. If an environment is located across different Resource Hosts they are connected using VXLAN tunneling.

##Management Traffic Traffic for communicating with the Management Host (MH) is isolated in special VLAN. By default it's configured to use VLAN tag 2. The default tag value can be overridden on all peer Resource Hosts in the VLAN configuration file: /var/lib/apps/subutai/current/vlan.

The Subutai Agent uses this management VLAN to communicate with the management server running on a MH container in one of the Resource Hosts of the peer.

##VXLAN VXLAN tunneling is used to connect containers in environments located on different Resource Hosts. VXLAN tunnels are managed by Open vSwitch connecting containers over a virtual switch: like a physical switch in a physical LAN.

##P2P The P2P network is another tunneling level that allows Resource Hosts to interact with each other assuming that all of them connected to the same LAN.

All NAT traversals, to connect NAT'd remote peers (without public IPs) is solved by this component.

P2P traffic is always encrypted using symmetric keys which change periodically: key changes are handled independently by another system. Traffic encryption is another network isolation mechanism.

If environment need to be created across several remote peers, Subutai will create P2P connection on each Resource Host. All of these P2P connections will be connected to the single swarm. Each Resource Host will get a new IP-address, to be used for tunneling traffic between Resource Hosts.

##Container Networking When containers interact with each other, they know nothing about their location. Inside an environment container to container traffic appears just like simple LAN traffic. Tunneling with VXLAN and P2P are lower level aspects of the implementation that containers in environments never see.

Each container has only one network interface, which is connected to OVS port using the environment's VLAN id.

All containers in an environment have unique static IP-addresses defined and managed by the Management Server.

If a locally created container is not associated with an environment, it will receive an IP-address from the DHCP server configured in each Resource Host. This container will get access to the Internet over the NAT. This container can later be reconfigured to participate in a defined VLAN.

The network configuration of containers are setup during the LXC build in the start/stop scripts. When starting containers the system checks if a virtual switch is configured for this container and if a VLAN is assigned to it. It then connects the container's port to the correct port on the virtual switch for the environment.

##SSH To Container Environment owner can directly access containers via SSH. This functionality is implemented using the following events across systems:

• Owners can join the environment network through a P2P connection on any host running the p2p daemon;

• Each Resource Host forwards traffic only from the p2p interface to the each container based on a special port;

• The SSH-server inside the container handles this SSH connection.

This is commonly used by the Subutai Tray Application for convenient SSH access. It can be installed on any desktop OS to make connecting uber easy. Users just choose which container he they want to SSH into via a pulldown on the system tray. It handles the p2p join into the environment and launches a terminal shell with the proper SSH command.