- I want to manage AWS infrastructure with YAML
- I want to use Kubernetes RBAC for authorization
- I want to be able to defines rules to govern my cloud resources
Why not :)
- For (1), there's AWS Controllers for Kubernetes (ACK) or maybe Crossplane
- For (2), it's quite straight forward. If we can express AWS resources using Kubernetes CRDs, then it's done.
- For (3), I'm thinking Kyverno or OpenPolicyAgent.
I will skip the part where you setup AWS CLI and an EKS cluster Suppose that is all set and done. If not, follow the brief instructions below to setup a new EKS cluster. The easiest way IMO is to use eksctl from Weaveworks.
aws ec2 create-key-pair --region ap-southeast-1 --key-name my-yaml-eks-key
eksctl create cluster \
--name my-yaml-eks \
--region ap-southeast-1 \
--with-oidc \
--ssh-access \
--ssh-public-key my-yaml-eks-key \
--managedWait for a bit for the cluster to be provisioned.
There will be 2 CloudFormation stacks being provisioned so it might take awhile. If something goes wrong (disconnection, etc..), you can check with this command below
eksctl utils describe-stacks --region=ap-southeast-1 --cluster=my-yaml-eks2021-10-18 22:35:28 [ℹ] eksctl version 0.70.0
2021-10-18 22:35:28 [ℹ] using region ap-southeast-1
2021-10-18 22:35:29 [ℹ] setting availability zones to [ap-southeast-1a ap-southeast-1b ap-southeast-1c]
2021-10-18 22:35:29 [ℹ] subnets for ap-southeast-1a - public:192.168.0.0/19 private:192.168.96.0/19
2021-10-18 22:35:29 [ℹ] subnets for ap-southeast-1b - public:192.168.32.0/19 private:192.168.128.0/19
2021-10-18 22:35:29 [ℹ] subnets for ap-southeast-1c - public:192.168.64.0/19 private:192.168.160.0/19
2021-10-18 22:35:29 [ℹ] nodegroup "ng-5c441b7d" will use "" [AmazonLinux2/1.20]
2021-10-18 22:35:29 [ℹ] using EC2 key pair %!q(*string=<nil>)
2021-10-18 22:35:29 [ℹ] using Kubernetes version 1.20
2021-10-18 22:35:29 [ℹ] creating EKS cluster "my-yaml-eks" in "ap-southeast-1" region with managed nodes
2021-10-18 22:35:29 [ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2021-10-18 22:35:29 [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-southeast-1 --cluster=my-yaml-eks'
2021-10-18 22:35:29 [ℹ] CloudWatch logging will not be enabled for cluster "my-yaml-eks" in "ap-southeast-1"
2021-10-18 22:35:29 [ℹ] you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-southeast-1 --cluster=my-yaml-eks'
2021-10-18 22:35:29 [ℹ] Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "my-yaml-eks" in "ap-southeast-1"
2021-10-18 22:35:29 [ℹ]
2 sequential tasks: { create cluster control plane "my-yaml-eks",
2 sequential sub-tasks: {
4 sequential sub-tasks: {
wait for control plane to become ready,
associate IAM OIDC provider,
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system/aws-node",
create serviceaccount "kube-system/aws-node",
},
restart daemonset "kube-system/aws-node",
},
create managed nodegroup "ng-5c441b7d",
}
}
2021-10-18 22:35:29 [ℹ] building cluster stack "eksctl-my-yaml-eks-cluster"
2021-10-18 22:35:30 [ℹ] deploying stack "eksctl-my-yaml-eks-cluster"Once it's done. Make sure the cluster is accessible
At first, I plan to use ACK but then I remember a friend of mine talked about Crossplane the other day so I want to give it a try. Bonus point, it works with multiple cloud providers :)
I'm going to use Helm so make sure you have it installed. Simply follow the official installation instructions on Crossplane website here.
As of this post, 1.4.1 is the latest chart version.
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --version 1.4.1NAME: crossplane
LAST DEPLOYED: Mon Oct 18 23:05:25 2021
NAMESPACE: crossplane-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Release: crossplane
Chart Name: crossplane
Chart Description: Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume.
Chart Version: 1.4.1
Chart Application Version: 1.4.1
Kube Version: v1.20.7-eks-d88609
Check the status
helm list -n crossplane-system
kubectl get all -n crossplane-system
Also, make sure to install the Crossplane CLI.
curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | shapiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws
spec:
package: crossplane/provider-aws:alphaAfter the provider is ready, apply the following next. You need to do this in 2 steps because otherwise, ProviderConfig kind is unknown.
apiVersion: v1
stringData:
config: |
[default]
aws_access_key_id = <your-aws-access-key-id>
aws_secret_access_key = <your-aws-secret-access-key>
kind: Secret
metadata:
creationTimestamp: null
name: aws-credentials
---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: aws-provider-config
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: aws-credentials
key: configCheck the status of the provider
kubectl get providers
Let's do a S3 bucket first. Make sure you do a random bucket name so that it won't conflict with an existing bucket.
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
name: test-bucket-1923981293819238
namespace: default
spec:
providerConfigRef:
name: aws-provider-config
forProvider:
locationConstraint: ap-southeast-1
acl: privateAfter a few seconds, the resource status will change to Ready and Synced.
So that's cool. What's next? Let's trying to use OPA or Kyverno to set some rules for our newly created resources.
I'm gonna go with Kyverno here. No particular reason. I just feel OPA is too mainstream. The concept with OPA is similar. You can take it as homework for your lab session.
Let's install Kyverno with Helm
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno-crds kyverno/kyverno-crds --namespace kyverno --create-namespace
helm install kyverno kyverno/kyverno --namespace kyvernoNow, let's write a simple policy. Say, we don't want to allow creating S3 bucket anywhere except in ap-southeast-1 region.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: allow-s3-ap-southeast-1-only
annotations:
pod-policies.kyverno.io/autogen-controllers: none
policies.kyverno.io/title: Allow creating S3 bucket in ap-southeast-1 only
policies.kyverno.io/subject: Bucket
policies.kyverno.io/description: >-
Allow creating S3 bucket in ap-southeast-1 only
spec:
validationFailureAction: enforce
background: true
rules:
- name: s3-in-ap-southeast-1-only
match:
resources:
kinds:
- Bucket
validate:
message: "Using any location other than `ap-southeast-1` is not allowed"
pattern:
spec:
forProvider:
locationConstraint: "ap-southeast-1"Now, let's try to create a S3 bucket in us-east-1 region. You will be blocked by the admission controller.
# bad-s3.yaml
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
name: test-bucket-091289310923801928309123
namespace: default
spec:
providerConfigRef:
name: aws-provider-config
forProvider:
locationConstraint: us-east-1
acl: privatekubectl create -f bad-s3.yaml
Now, the policy we have here is very simple but you can do all kind of stuff with Kyverno cluster policy. It would be a cool weekend hack to convert all rules you have currently to Kyverno cluster policy. That sounds like tons of fun :)
I'm not going to tell you should start doing this but it's a feasible way of managing infrastructure at small scale.
Also with the release of AWS Cloud Control API, the resources model is very close with the Kubernetes resources model now. I'm expecting ACK to get much much better.
Have fun hacking AWS :)