Kubernetes Security Best Practices. Hardening Kubernetes Cluster (1 of 3)

As organizations increasingly adopt containerization and microservices, Kubernetes has emerged as the go-to solution for orchestrating container deployments at scale. The open-source platform automates the deployment, scaling, and management of containerized applications, making it an invaluable tool for modern DevOps teams. However, as with any technology that deals with sensitive data and mission-critical applications, securing Kubernetes environments is of utmost importance.

The growing popularity of Kubernetes has inevitably attracted the attention of cybercriminals, who constantly look for vulnerabilities and misconfigurations to exploit. Consequently, it is crucial for businesses and developers to take a proactive approach to security, ensuring that their Kubernetes clusters are protected against potential threats.

Kubernetes security topic is very multifaceted topic. Several books will not be enough to cover the subject comprehensively. This article aims to provide an overview of best practices for securing Kubernetes cluster. While describing best practices related to the core architecture elements, we will focus on some of the most common and crucial aspects of setting Kube API server, kubelet and ETCD storage. We won’t discuss there securing of deployed applications, network security or role based access. That will be the topics for the future articles.

• Hardening Kubernetes Cluster

• Hardening Application Environment

• Enhancing Runtime Security and Policy Enforcement

Kube API security features

The Kubernetes API server is the central component of a Kubernetes cluster, responsible for exposing the Kubernetes API and processing incoming requests. Ensuring the API server is secured and configured correctly is essential for overall cluster security.

We can set some configuration keys to make it more secure. Depending on the way you have deployed Kubernetes, you should edit either kube-apiserver.service file or kube-apiserver manifest to add some security flags. By default, the API server allows anonymous requests. Disabling anonymous authentication ensures that all requests must be authenticated, reducing the risk of unauthorized access.

So, you should add --anonymous-auth=false. Setting some value for --request-timeout=<duration> flag may help to protect from DoS attacks limiting the time spent on processing request. --encryption-provider-config=<path> enables the encryption of data in etcd and provides the path to a configuration file specifying the encryption provider and keys to be used. We will talk about storing data in etcd in a while. This is not an exhaustive list of flags for the api server, but these one you definitely should use in your cluster.

Kubelet security options

Another important part of inter cluster communication is kubelet, which provides communication of worker-nodes with master-nodes via kube API. By default kubelet allows all unauthenticated requests. I means that everybody can access to all resources located on the worker node. It is a great security risk. To prevent it we should add --anonymous-auth=false to kubelet.service file or add the section authentication:anonymous:enabled:false to the kubelet-config.yaml file.

And enable one of authentication mechanism such as certificate based or API Bearer Tonkens. To enable certificate based authentication you should add path to client ca certificate in --client-ca-file=<path-to-certificate-file> flag in kubelet.service file or authentication:x509:clientCAFile:<path-to-certificate-file> section in kubelet-config.yaml file.

If you provide authenticated access to kubelet API, don’t forget to add --kubelet-client-certificate and --kubelet-client-key flags in the settings of kube api as far as it kube api service is a client of kubelet api service. Ok, we have got rid of unauthenticated requests, but that is not enough.

By default kubelet API server allows every request to every authenticated user. That means even if some user is not allowed to have access to pods in particular namespace via kube API, for example, he can reach them through kebelet API service on the node where this pods are located.

To prevent it we should set --authorization-mode=Webhook kubelet.service file or authorization:mode:Webhook section in kubelet-config file. In such case kubelet will ask kube API server is the user authorized to make such request. And tone more thing about kubelet. In some installations it allows all anonymous read-only requests to http://localhost:10255/metrics url even if you explicitly set --anonymous-auth=false. It made for some external monitoring tools, but is is not good idea to use it. To disable it just set --read-only-port=0 flag in kubelet.service file or readOnlyPort:0 section in kubelet-config.yaml file.

ETCD encryption

As we know Kubernetes stores all data in etcd. By default all data in etcd are stored in plain text format. What does it mean in terms of security? I means that anyone who get access to etcd host cat read all data related to kubernetes resources, include secrets.

You can run:

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/default/secret_name | hexdump -C

and you will see secret data in plain text.

So, the good idea to encrypt secret resources in etcd. To do that we should create a config file in yaml format which can be look like this

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

Several comments according to this file. In section resources:resources we define which types of resources should be encrypted. Generally, you may encrypt all data, but it is not a good idea, as far as encryption and decryption operations are rather resources hungry actions, and it will decrease performance of your cluster.

You should want to encrypt some sensitive data such as secrets and, maybe, configmaps. In the resources:providers section we have the ordered list of encryption providers to use. Only the first provider encrypts newly created resources, and all of them may be used to decrypt the date, also in order they are mentioned. Identity provider means no encryption, so, you should place it on the bottom of the list. If you change encryption provider for some reason, you should leave the old one below in the list to make etcd be able to decrypt early created resources.

After you created encryptionConfiguration file, you should add --encryption-provider-config=<path> flag to kube api configuration as we mentioned above, and also, you should mount it to kube-api pod (if you use container based installation of Kubernetes). After encryption is apllied, all new resources will be stored in encrypted format, but not old ones.

To encrypt early created secrets you should recreate them in a way like this kubectl get secrets --all-namespaces -o json | kubectl replace -f - If you need to disable encryption, just put identity provider to the first position in your encryptionCоnfiguration file.

Remember one important thing. This operation is encrypts data in etcd, not in kubernetes, and you will still be able to get secrets data in base64 encoded format. So encryption etcd data doesn’t exempt from limitation access to secrets resources using RBAC methods.

Conclusion

While securing a Kubernetes cluster can be complex, the strategies and guidelines discussed in this article provide a solid foundation to harden the Kubernetes API server, Kubelet, and enable etcd encryption. Ensuring these key components are secure is a crucial step in fortifying your Kubernetes environment against potential threats.

However, no security strategy should be considered complete without regular reviews and audits. One useful tool in this regard is kube-bench, an open-source tool that tests a Kubernetes cluster against the Center for Internet Security (CIS) benchmark. The CIS benchmark is widely accepted as a reliable standard for Kubernetes security, providing a comprehensive checklist of security best practices. By running kube-bench, you can identify potential security risks and receive recommendations for hardening your cluster based on the CIS benchmark.

Remember, security is an ongoing process that requires continuous attention and improvement. This article is just a starting point in your Kubernetes security journey. We will continue to explore other aspects of Kubernetes security in future articles, aiming to provide comprehensive insights and practical guidelines to help you secure your Kubernetes environments effectively.

Thank you for joining us on this journey. We look forward to further exploring the world of Kubernetes security with you. Stay tuned for our upcoming articles, where we will delve deeper into the crucial aspects of Kubernetes security.