Change applications in platform from being deployed as daemon sets to deployments

[SandGrainOne]

Description

The purpose of the change is to have autoscaling and improve utilization of the AKS resources.

Additional information

The platform team is working on a base template for platform applications: Altinn/altinn-platform#978

We also hope that this can have a positive effect on the liveness probes and kubernets ability to recyle pods that are unresponsive. There has been cases where Profile has stopped responing on 1 out of 3 requests. We believe it's because one out of three pods across the three nodes in AT was faulty.

Profile appear unresponsive in AT and TT02

I managed to recreate the issue using Postman against AT23. I was getting consistent results where 2 requests worked fine and one request failed. With three nodes there are 3 pods because of daemon sets.

Restarting all instances of the Profile application in the environment had a positive effect. Decided to perform the same restart in all AT environments and in TT02.

Follow up:
Is it the health check that isn't good enough or is there something else going on?

Tasks

• Make all necessary changes to move Profile over to deployments
• Ensure labels are set properly for deployments so it can be picked up by KubernetesWrapper
• Define pod scaling limits
• Pod disruption settings. "Run at least one pod of every application"
• Affinity to have the pods for an application spread across nodes

Acceptance Criterias

• Altinn Profile runs as deployments and not as daemon sets.

[olebhansen]

@SandGrainOne : is this actually being solved as part of #190 (and thus this issue can be closed as duplicate/"superseeded")?

[SandGrainOne]

@olebhansen yeah, that is the intention.

[olebhansen]

Closing. Duplicate of #190.

[SandGrainOne]

I reopened this to solve a part of issue #190 . #190 will be postponed slightly in favor of #214.

[SandGrainOne]

It turns out that this type of change also will require a blue-green deployment process. When deploying the changes to kubernetes, it tears down the existing daemon set before starting new pods with the new settings.

This puts a big blocker on this change at least for now. I want us to plan a larger update more of our applications instead of doing a blue green change for every "little" thing.