These instructions modify nodes in your cluster. Only run on test clusters currently.
The following instructions install the Trow registry on an existing Kubernetes cluster, with a
certificate signed by the Kubernetes CA. They have been primarily tested GKE. Minikube should work, but there may be issues with signing certificates. The instructions assume kubectl is configured to point to your cluster.
We recommend spinning up a small GKE cluster for testing Trow to begin with.
MacOS warning: I've just realised that some of the Bash scripts don't work on MacOS. I'll update shortly.
- If you're running on GKE or have RBAC configured you may need to expand your rights to be able to create the needed service-account (on GKE the user is probably your e-mail address):
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<user>
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" created
- If you've not already done so, clone the Trow repository:
$ git clone https://github.com/ContainerSolutions/trow.git
...
- Apply the
trow.yamlfile from the root of this repository:
$ cd trow
$ kubectl apply -f trow.yaml
serviceaccount "trow" created
role.rbac.authorization.k8s.io "trow" created
clusterrole.rbac.authorization.k8s.io "trow" created
rolebinding.rbac.authorization.k8s.io "trow" created
clusterrolebinding.rbac.authorization.k8s.io "trow" created
deployment.apps "trow-deploy" created
service "trow" created
- This will create a service for Trow that includes a NodePort for external
access (if you don't want this, edit
trow.yaml). It will also pull the Trow image and start up the pod, which may take a moment to download. The Trow pod will then get stuck in init, waiting for us to approve its certificate. Do this by:
$ kubectl certificate approve trow.kube-public
certificatesigningrequest.certificates.k8s.io "trow.kube-public" approved
- If you get the error "No resources found" wait a moment and try again. In some cases it takes a few minutes for the request to appear in k8s.
- Trow should now be up and running, but we still need to make the nodes trust
the certificate if we want them to be able to pull. The easy way is by
running the following script, but be aware that this will modify files on the
Nodes, including
/etc/hosts:
$ cd install
$ ./copy-certs.sh
Copying certs to nodes
job.batch "copy-certs-5a2fa2bc-3457-11e9-a2bc-42010a800018" created
job.batch "copy-certs-55cf8134-3457-11e9-a2bc-42010a800018" created
Note there is an issue with this approach, as new nodes will not automatically get the certs and will be unable to pull from Trow. We will fix this issue in the future.
- Finally, you probably want to be able to push images from your development laptop, which you can do with:
$ sudo ./configure-host.sh --add-hosts
Copying cert into Docker
Successfully copied cert
Adding entry to /etc/hosts for trow.kube-public
Exposing registry via /etc/hosts
Successfully configured localhost
This will copy Trow's cert into Docker and also add an entry to /etc/hosts for trow.kube-public. We can test it all out by trying to push an image:
$ docker pull nginx:alpine
alpine: Pulling from library/nginx
Digest: sha256:e0292d158b6b353fde34909243a4886977cb9d1abb8a8a5fef9e0ff7138dd3e2
Status: Image is up to date for nginx:alpine
$ docker tag nginx:alpine trow.kube-public:31000/test/nginx:alpine
$ docker push trow.kube-public:31000/test/nginx:alpine
The push refers to repository [trow.kube-public:31000/test/nginx]
979531bcfa2b: Pushed
8d36c62f099e: Pushed
4b735058ece4: Pushed
503e53e365f3: Pushed
alpine: digest: sha256:bfddb36c23addfd10db511d95b7508fa7b6b2aca09b313ff3ef73c3752d11a55 size: 11903
If the push seems to hang, check if port 31000 is blocked (common in GKE).
The Kubernetes cluster should now be able to pull and run the image:
$ kubectl run trow-test --image=trow.kube-public:31000/test/nginx:alpine
deployment.apps "trow-test" created
$ kubectl get deploy trow-test
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
trow-test 1 1 1 1 8s
One of the major features of Trow is the ability to control the images that run in the cluster. To achieve this, we need to set-up an Admission Webhook in the Kubernetes cluster that will ask Trow for permission everytime a resource is created or updated.
The default policy will allow all images local to the Trow registry to be used, plus Kubernetes system images and the Trow images themselves. All other images are denied by default, including Docker Hub images.
To enable validation run (from the install directory):
$ ./validate.sh
Setting up trow as a validating webhook
WARNING: This will limit what images can run in your cluster
validatingwebhookconfiguration.admissionregistration.k8s.io "trow-validator" configured
Now try running a Docker Hub image, which should be denied:
$ kubectl run proxy --image=docker.io/nginx
deployment.apps "proxy" created
$ kubectl get deployment proxy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
proxy 1 0 0 0 13s
$ kubectl describe rs proxy-<TAB>
...
Warning FailedCreate 16s (x13 over 57s) replicaset-controller Error creating: admission webhook "validator.trow.io" denied the request: Remote image docker.io/nginx disallowed as not contained in this registry and not in allow list
But local images still run:
$ kubectl run local-proxy --image=trow.kube-public:31000/test/nginx:alpine
deployment.apps "local-proxy" created
$ kubectl get deploy local-proxy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
local-proxy 1 1 1 1 16s
If you want to allow images from the Docker Hub, take a look at the --allow-docker-official and --allow-prefixes arguments. This can be passed to Trow via the trow.yaml file.
At this time the only authentication available is a simple username & password combination that can be set when starting Trow. To enable this, use the -p and -u arguments, which can be set in the appropriate section of the trow.yaml file:
...
containers:
- name: trow-pod
image: containersol/trow:default
args: ["-u", "myuser", "-p", "mypass", "-n", "trow:31000 trow.kube-public:31000"]
imagePullPolicy: Always
...
After this you will need to run docker login to push and pull images:
$ docker pull trow.test:8443/myimage
Using default tag: latest
Error response from daemon: Get https://trow.test:8443/v2/myimage/manifests/latest: unauthorized: authentication required
$ docker login trow.test:8443
Username: myuser
Password:
Login Succeeded
$ docker pull ...
Trow also accepts a pointer to a file containing the password via the --password-file argument instead of -p. This allows the password to be stored in a Kubernetes secret that can be mounted into a volume inside the container.
- If you get an error when pushing, check the logs for the Trow pod e.g:
$ kubectl logs trow-deploy-5cf9bccdcc-g28vq -n kube-public
...
- If a deployment isn't starting, check the logs for the replica set e.g:
$ kubectl get rs my-app-844d6db962
...
- If there is a failed create message, the image may have been refused validation by Trow. If the message reads like:
Error creating: admission webhook "validator.trow.io" denied the request: *Remote* image docker.io/nginx disallowed as not contained in this registry and not in allow list
That means Trow considered the image name to refer to a remote repository (i.e. not Trow itself) which has not been added to the allow list. If you believe the image should have been considered local, check the repository address appears in the list of addresses passed to Trow on start-up with the -n switch. If you want to allow a single remote image, add it to Trow by using the --allow-images flag. If you want to allow a whole repository or subdirectory of a repository use --allow-prefixes.
- If the message reads:
Error creating: admission webhook "validator.trow.io" denied the request: Local image trow.kube-public:31000/notpresent disallowed as not contained in this registry and not in allow list
It means Trow expected to be able to serve this image itself but it wasn't found in the repository. Either push the image or use the allow-images or allow-prefixes flag to pre-approve images. Note that Kubernetes will keep trying to validate images.
- If you get the error:
Error creating: Internal error occurred: failed calling admission webhook "validator.trow.io": Post https://trow.kube-public.svc:443/validate-image?timeout=30s: no endpoints available for service "trow"
Kubernetes has probably tried to start a new instance of Trow, but can't because there is no Trow instance to validate the image (can you say "catch 22"?). This will largely go away when we have HA, but until then you'll have to disable validation for Trow to restart. You'll also have to repeat the install steps for approving the certificate, distributing the certificate and setting up validation (in the future we will look to automate this process or reuse certificates to simply this).
- If you get errors about certificates, either in
docker pushor from a replica set, you may need to re-approve and distribute the certificates (possibly due to the Trow pod being restarted):
$ kubectl certificate approve trow.kube-public
$ ./copy-certs.sh
$ ./sudo ./configure-host.sh --add-hosts
$ ./validate.sh
See above for full details.
Trow can be run with the --no-tls flag to serve over HTTP only. This can be
useful in development and testing, or when running on an internal, secure
network where the risks are understood.
The major problem is that the Docker client will not by default allow images to be pushed or pulled without TLS. This can be circumvented in two ways:
-
Using the localhost address for the registry.
-
By adding an "insecure-registries" entry to the Docker
daemon.jsonfile. https://docs.docker.com/registry/insecure/
Method 1) can work well internally in a cluster using NodePort to forward traffic. Method 2) can then be used to get an image into the registry from a development machine.