Strimzi without self-managed CA certificates

[DavidMachacek]

As of this moment Strimzi support providing custom CA, i.e.

spec:
  clusterCa:
    generateCertificateAuthority: false
  clientsCa:
    generateCertificateAuthority: false 

These certificates are expected to be CA:True

X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE

But our internal security policy prevents us from granting Strimzi or anyone else the private key from CA certification chain that includes our Root and intermediate CA. Why does it need it? We just want strimzi to be able to validate signed certificated, which we would prefer to provide by our own to each cluster member (zoo, brokers..) as well as to each client. Is this achieveable/configurable in Strimzi?

Our expectation is, that for example if Strimzi support providing custom clients certificates along with external-tls authentication, it should provide an option to run everything using CA managed by us, not by Strimzi itself.

[scholzj]

For the Cluster CA, Strimzi uses the CA to generate the server certificates for the various components. For that, we need you to provide a CA including its private key as that is needed to issue the server certificates. That said, if you use some complex setup with CA hierarchy consisting of multiple CAs, you do not need to provide the private key for each of the CAs. Just for the CA that should be used to sign the certificates. So if you have hierarchy of 3 CAs - Root CA -> Intermediate CA -> Cluster CA - you need to provide the private key only for the Cluster CA.

For the Clients CA, the CA is used to sign the user certificates. If you want to use the User Operator to issue user certificates, you need to provide the CA with the private key (again, only for the Clients CA, not for any Root CA that signed the Clients CA). If you do not plan to issue any user certificates through the User Operator, you can provide something dummy for the private key and only use the real public key.

[DavidMachacek]

Our organization would not issue a CA certificate with hierarchy that includes root CA, because everything signed by this cert, would also be also trusted by the rest of the organization outside of kafka environment. This is especially considered as a threat since the cluster CA must contain valid keys in order to sign certs for brokers.
We want to keep our environment both

• contained - you cannot generate certificates valid outside of strimzi environment
• trusted - so everyone could check, it is really us.

But with recommended setup everyone with access to strimzi namespace secrets (i.e. kafka admin), could potentionally generate certificates valid outside of the cluster using cluster CA keys.
But if we use different certificate chain coming from different Root CA, it would compromise the truststores (there should be just one Root CA for our organization).

I wonder what is the best practice, how other organizations approach this?

1. shall we create a new intermediate CA in order to be able to revoke it, if any incident occurs?
2. For example if I wanted to use standard signed valid certificate from other authorities (i.e. LetsEncrypt), they would never provide me with a CA Cert, thus it would not be usable for this setup
3. even if I could hide the keys (i.e. you dummy for clients, the security will still struggle to provide us with the CA anyway
4. is it feasible to configure a setup where everything is issued by Intermediate CA and the cluster and clients certs are used ONLY to validate signatures, but never to sign other certificates? That seem to be an optimal scenario from security point of view!

[scholzj]

Our organization would not issue a CA certificate with hierarchy that includes root CA, because everything signed by this cert, would also be also trusted by the rest of the organization outside of kafka environment. This is especially considered as a threat since the cluster CA must contain valid keys in order to sign certs for brokers. We want to keep our environment both

I think the question is why do you want to use your own CA when it sounds like you actually do not want to use your own CA? Of course, everything has its own consequences and trade-offs. You have to evaluate them and decide what is the right approach for you. I guess the main question to answer here is why do you want to use the custom CA in the first place.

But with recommended setup everyone with access to strimzi namespace secrets (i.e. kafka admin), could potentionally generate certificates valid outside of the cluster using cluster CA keys. But if we used different cert chain coming from different Root CA, it would compromise the truststores.

I'm not sure what exactly you mean with the recommended setup here. The general expectation is that you have some security in place to allow access to the namespace only to selected people. Those people can always do a lot of bad if they decide to. So can any kind of administors. So this risk of course exists and in general is hard to eliminate completely in any situation.

I wonder what is the best practice:

1. shall we create a new intermediate CA in order to be able to revoke it, if any incident occurs?

You can certainly use a separate CA hierarchy to separate the Kafka certificates from other certificates you use for other systems. But I think you should be careful about how well the certificate revocation really works -> from my experience many clients simply ignore any revocation lists. So it is usually not as simple as revoking something. Using certificates with shorter validity can help - but that means more work as you need to renew them more often.

2. how other organizations approach this? For example if I wanted to use standard signed valid certificate from other authorities (i.e. LetsEncrypt), they would never provide me with a CA Cert, thus it would not be usable for this setup.

From my experience, different organizations deal with this differently because they have different requirements and processes. Security in general is always heavily opinionated and what one organization wants to do because they see it as best security practice is often a no-go for another organization. TBH, that makes it quite hard for us as well because there is no single way to do things that we can follow.

As for the public CAs such as Let's Encrypt - that is a misunderstanding of how they work. You cannot use them for the Cluster CA or the Clients CA and they were never designed to be used that way. They are not able to issue certificates for user authentication and they are not able to issue any certificates for internal hostnames used within the Kubernetes cluster - this is what the Cluster CA does.

If you want to use Let's Encrypt (or other public CA) with Strimzi, you can use it for the custom listener certificate with external Kafka listener that uses some public DNS names. The benefit you get from it is that you do not need to distribute any special truststores to your clients etc. But overall, this is a very different use-case from custom CA.

3. is it feasible to configure a setup where everything is issued by Intermediate CA and the cluster and clients certs are used ONLY to validate signatures, but never to sign other certificates? That seem to be an optimal scenario from security point of view

Sorry, I'm not sure I really follow what do you mean by this.

[DavidMachacek]

Sorry, I'm not sure I really follow what do you mean by this.

Is there a way to configure Strimzi not to sign any certificates? But to only validate other signatures? Let me explain..

OPTION 1:

In this setup we would create all the certificates for both cluster entities (brokers/zookeepers..) and client entitites (producers/consumers..) manually. This way there would be no need to provide any CAs or keys to Strimzi itself. All the certificates would be signed by central secured certificate issuer (i.e. managed by security admin) and all the certificates will be used just to validate each other using public key infrastructure (i.e. openssl verify -CAfile root-ca.crt client.crt).

I tried to draw a diagram below to better explain what i mean:

OPTION 2:

Make some sort of hybrid solution, where we issue a self-signed Cluster CA for signing cluster resources. But at least for the clients part we would like to keep our own organization Root certificates. We dont really care about what certificates Strimzi resources use to communicate with each other, but all the client facing component should have our Organization Root certificate.
Please see the diagram below:

Thanks for the answer, we really appreciate your time

[scholzj]

Re 1) In theory, if you pre-create the secrets with the certificates you issued your self in whatever way they will be just re-used. But you will need to use exactly the same subjects and SANs as Strimzi uses to make sure they would work. I have also no idea how the renewal would work for these. So you would need to reverse engineer it I guess.

Re 2) Sure, you can do this. But for the cluster CA, I'm not really sure what is the benefit you expect from it.