Proposal: Armada operator

[jimbobby5]

This is a proposal to create an Armada operator.

Reference for the operator pattern: https://kubernetes.io/docs/concepts/extend-kubernetes/operator

An example implementation: https://github.com/prometheus-operator/prometheus-operator

Armada comprises a number of components that need to be installed and configured correctly in order for the wider system to function. Furthermore the system requires ongoing maintenance to ensure that its components remain healthy. There is also application-level state that needs to be configured and controlled (e.g. queues, namespaces, etc).

It has been observed that not having a clear and simple way to install/configure Armada in a Kubernetes environment is a hindrance to open-source adoption. The standard approach for applications of anything more than simple complexity is to develop an operator (which itself is simple to install) which is responsible for the actual deployment and management of the application components, based off simple some CRDs.

This proposal is to create an Armada operator which will manage the installation of the Armada components on a given Kubernetes cluster. This will solve the problem of not having a simple installation pattern. Later, it can be extended to handle ongoing management of the Armada components, and configuration of the system.

Basic steps:

• Create a design for an operator for 'installing' Armada. This would include (but not be limited to):
  • Definition of required CRDs (e.g. ArmadaServer / ArmadaExecutor / ArmadaLookout ...)
  • Overview of internal structure of operator
  • Definition of control loop flow
  • Any non-functional requirements (e.g. runtime environment / performance / extensibility / ... )
• Agree design of operator
• Implementation

Later, we can repeat the process to extend the operator in phases to:

• Maintain system health and readiness
• Handle upgrades
• Configure required Kubernetes structures (e.g. namespaces for users)
• Configure required Armada structures (e.g. queues)

[severinson]

I like it. I think this sounds like where we'd want to be in the long-term.

A possible way of getting there:

1. Improve setup/initialisation. Identify ways in which we can change Armada to make setup/installation easier and create helm charts for the actual setup/initialisation.
2. Identify ways in which monitoring Armada can be made easier. Create the operator.

I think 1. would already provide a lot of value and is easier than 2. Further, I think 2 will be made easier by already having done 1. so the effort would not be wasted even if we don't use helm in the long-term.

Finally, I'm sure we'll find many ways in which setting up/initialising/monitoring Armada is harder than necessary. Identifying these early so that we can make improvements to Armada would likely simplify the operator.

[jimbobby5]

I'd prefer to shoot for creating the Operator to start with. I don't think the system is complicated enough already that it can't be set up with an operator

[Sharpz7]

Do we see this going in this repo or a seperate one?

[jimbobby5]

I don't personally have a strong opinion on this

[dejanzele]

I recently played with EFK stack, and I used the Elastic Operator to install all of the compontents. It is an operator which handles installation, configuration and component wiring, and then it exposes CRDs for ElasticSearch, Kibana, Filebeat...

Pretty much the exact same thing we want to accomplish. It works really elegantly and is really straightforward, maybe we can steal a pattern or two :)

You can find more info here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html

[Sharpz7]

I think fundementally the thing that seperates this from helm is an operator can make upgrading significantly easier, as well as having better self-healing mechanisms for data and cross-pod operation.

Would we want CRD's for setting up databases as well? Or would it just be the main armada components (lookout, server, ingester etc)