[WIP] Hardware requirements for Kubernetes cluster

[imilchev]

In this discussion I will try to list the minimum hardware requirements for the platform. Since our cluster will consists of a limited number of nodes the requirements for the control plane are not that strict.

Master nodes

The master nodes run etcd and the Kubernetes control plane. I was not able to find specific requirements from any official sources apart from the etcd website. They suggest 2 cores and 8GB of RAM for clusters of up to 50 nodes (see here). However, our clusters will be nowhere near 50 nodes so I would assume that less memory should suffice. Some managed Kubernetes platforms like Kublr suggest that it is enough to run the control plane on 2 cores and 4GB RAM (see here). A few blog posts suggested the same so I would expect 2 cores and 4GB RAM to be more than enough.

Storage

The storage requirements of the master nodes are minimal. They only need to run a handful of containers which do not demand excessive storage. A disk of around 30-40GB should be more than sufficient.

Worker nodes

The worker nodes are more demanding as they are running all applications required by the platform. I will try to make a list of the applications we rely on excluding our own modules and list their requirements so we have an overview of that

Ghost

The CMS system that we currently use. On its website it mentions that it requires at least 1GB RAM and I would assume 1 core is more than enough as it is not explicitly mentioned.

Monitoring and alerting

Currently, we are using the rancher monitoring and alerting app that is a package containing Grafana, Prometheus, node exporters and alert manager. According to the official documentation. The app requires ~2.5GB RAM and ~1.5 cores.

Storage

The recommended initial storage is 50GB. That should be aligned with the retention periods we define for the metrics.

Keycloak

The current identity provider that we use. According to its documentation it requires 512MB RAM and 1 core.

Storage

It requires at least 1GB or storage.

On top of that it depends on a database which at the moment we deploy separately just for Keycloak. However, it is the same type of database (PostgreSQL) that we require for our own modules, so for the sake of simplicity we should merge the 2.

PostgreSQL

Different sources suggest minimum requirements of 2GB RAM and 2 cores (for example, see here). However, we have multiple modules using the database + Keycloak so at least doubling the RAM seems realistic.

Storage

It really depends on the amount of data we need to store but a safe guesstimate would be ~10-15GB.

Ceph

The distributed storage provider that we use. It will be provisioning the persistent volumes of the cluster and providing an S3-compliant storage for storing campaign-related files. Ceph consists of multiple components which I will try to summarize below (info is based on SUSE recommendations described here:

• MONs - they try to maintain memory usage of around 1GB RAM, however that can go up in failure scenarios. SUSE recommends 4GB RAM and 4 cores.
• MGR - the more modules, the more memory is used. A recommended starting value is 2GB RAM and 2 cores.
• OSDs - 1 core (they mention 2GHz)

Summary

A summary of all requirements is provided below:

Application	CPU cores	GB RAM	GB Storage	Per worker?
Ghost	1	1	5	no
Monitoring	1.5	2.5	50	0.1 cores/50MB RAM
Keycloak	1	0.5	1	no
PostgreSQL	2	4	15	no
Ceph MON	4	4	-	no
Ceph MGR	2	2	-	no
Ceph OSD (per disk)	1	4	-	per disk
Total	12.5	18	71	for 1 worker

Proposed cluster setup

Type	Amount	CPU cores	GB RAM	System disk	Ceph disk
master	1	2	4	40	-
worker	2	8	16	40	200

[igoychev]

Perfect and well explained what are the needs and the reasoning about why!
My only concern, which doesn't need to be solved now is that Ceph(the file storage) is taking more than half of the total CPU/RAM. I guess this will self-balance itself when the need comes to add more App/DB worker nodes in future...