Hello, friends. thanks for joining my talk. let's see how we can master Kubernetes, security. I'm Deepu K. Shashidharan. I'm a staff developer advocate at Okta. I'm also an open source, enthusiast and a polycloud developer. I co lead the Jhipster project. I have built a Kubernetes dashboard called Kdash. you can, find out more about me on my website, deepu. tech, or you can find me on BlueSky at, deepu105. So let's, get into the talk. So before we talk about, security best practices in Kubernetes, I think it is important for us to have a basic understanding of Kubernetes security. Like any other complex piece of software, security in Kubernetes is multifold. So it can be broadly categorized into four layers. so first we have, the transport layer security, which is insured using, TLS. So all a p communication is done via TLS using valid certificates that forms the base layer of security. Then we have the authentication layer, and there is a flexible authentication layer with multiple supported mechanisms, including open id, connect, token based authentication, and so on. So all a PA requests are authenticated with one of these, several authentication mechanism. Then we have the authorization layer. And, again, all authenticated requests are authorized using one or more of the supported, authorization models. We'll talk a little bit more about this, later on. And finally, we also have a possibility of adding custom admission control modules to further lockdown access. which means all authorized requests, except read and get requests, are also validated by admission control modules. So Kubernetes comes with many security options out of the box, but to bulletproof your infrastructure, you need to consider many more security best practices. Today, we look into some of the vital security best practices. You can also find a blog post from me, with this content on the link provided here. Most of the security best practices, are not enabled by default. So you would have to do this yourself. And, most of these, are extremely important. So let's, get in and see what they are. first and foremost would be using RBAC. Kubernetes supports multiple, authorization models for its, API server. so we have, ABAC or Attribute Based Access Control. We have RBAC or Role Based Access Control. We have Node Authorization and Web Mode. out of all these, RBAC, I would say, is the, most, secure option. It's also the most widely used, option and it is ideal for enterprises and large to, medium to large organizations. So with the role based access control, you can define, a control structure that closely resembles your organization's business rules or access structure that you have. based on business, rules. So RBAC also works great with, OIDC authentication, and it is easier to manage if you have, an OIDC provider. RBAC makes it easier as well. using RBAC would be the number one, security best practice. So most Kubernetes distributions have RBAC enabled by default. So you can check this by running the kubectl command, kubectl cluster info dump, and then look for the authorization mode, in the output. ideally, it should show authorization mode equals RBAC. if not, you can enable it using the, authorization mode flag for the API server, while creating or patching the cluster. For example, setting the authorization mode, flag to RBAC comma node will enable both RBAC and node authorization on the cluster. Once RBAC is enabled, you can create, roles, and, role bindings for either, role or cluster role, or role binding or cluster role binding to control access to your resources. Here's an example of a role and role binding that lets users view ports and services. I would recommend checking out the Kubernetes documentation on our back for, for the learning. this is a very simple ex, example. You can do much more with, role and, role bindings. you can, control, who has access to what, on a namespace, level or at the resource level, for example. So next security best practice would be to use OpenID Connect for authentication. So Kubernetes supports many authentication mechanisms. Some of the most common ones are like client certificate based authentication, basic authentication, token based like service account tokens, bearer tokens, and so on, proxy based authentication, and finally OpenID Connect based authentication. So out of all these authentication mechanisms, OpenID Connect or OIDC is the most secure and scalable solution. So it is ideal for clusters accessed by large teams as it provides a single sign on solution for all users and makes it easy to onboard and offboard users. It is also way more secure than other mechanisms as you don't have to store any sensitive information on a user's computer, like client secrets and passwords. You can also use, advanced features like multifactor authentication, pass keys, hardware, tokens, like UB keys, et cetera, as most of IDC providers would be, might be supporting, these features as well. and YDC with R Bag gives you powerful access control that can be managed in, Identity systems like Auth0 or Okta. for example, if you are using, OpenID Connect along with, Okta, or Auth0, for your Kubernetes cluster, you might, you will be able to use all the advanced security features provided by these platforms. You would be able to configure, and manage users in these systems instead of, meddling with the Kubernetes installation. You'll be able to manage your RBAC setup, with these platforms. So it gives you a lot more flexibility and it is ideal for larger teams. Here is how UIDC works with communities. When you run a command with, cube, CTL, it uses, cube login, and opens a browser to authenticate you with your OIDC provider. once, you authenticate, with your YDC provider, let's say, or zero in this case, you will get, authentication response, and the odd response, the, the cube login will get no author response, from the browser. It'll then use that, response to make a token request to your application server to get a access token. And then it'll use that, it'll pass that access token to Cube CTL, which can use that token to make, a PA requests here Banes cluster. This makes the entire, setup extremely secure and, easy to, control. If you are interested in setting this up, you can check out this blog post that I wrote some while ago. It shows how to set up OpenID Connect on a Kubernetes cluster and how to use Okta to manage this. one of the most often overlooked aspects of security in Kubernetes is secrets. And that's why, next best practice would be to use secure secrets. So many don't set up proper secret management and end up using stock Kubernetes secrets, which are not technically secrets. because, the default Kubernetes secret is nothing but, plain text, resource, with, base64 encoded strings. And base64 encoded is not same as encrypted. So your secrets are in plain text and anyone, can, decode base64. And you don't need any, private keys or anything to decode base64, because it's not encrypted. which means you can't store them in your version control, making GitOps difficult. And you will end up resorting to environmental variables and stuff like that, unnecessarily complicating your setup. Even worse would be accidental commit of secret definitions, along with Kubernetes manifest to public Git repos in case of open source software or source available, projects. And other disadvantages are that Kubernetes secrets does not work with external secret managers. we have, multiple options, here, to work around this issue. one of the option would be to use a sealed secrets. sealed secrets are simplest to use as it, encrypts, the secrets on developer machine, which can be decrypted only by the cluster when it, will be used. This makes it possible to store, the secrets along with other data in your version control as they are encrypted. This is ideal for smaller setups and smaller teams. Here is how it will work. So you deploy a seal secret, controller. In your, cluster, the controller generates a private public key pair. the developer, has access to, developer can get the public key out of, this, generally extracted out. the private key is kept, secret and, you should, have proper RBAC on this, resources. So that's not exposed. the developer then uses the public key extracted. from the cluster to encrypt any sensitive, data, in their machine using the kubeseal CLI. for example, if you have a secret YAML, you can, encrypt that using this public key and make it into seal secret, YAML. the encrypted sealed secret resource can now be committed and deployed to the server. to the cluster. the sealed secret controller. so when you, this is where you will put it in the git repository. It is safe because it is encrypted now. when you apply or deploy this to your cluster. the sealed secret controller will, scan for any sealed secrets and when it detects one, it will use the private key, stored on the sealed secret controller to decrypt the sealed secret and it will create a normal Kubernetes secret with the, basics differ encoded, secret. It's simple and effective mechanism. Of course, you would still need RBAC to restrict access to the created Kubernetes secrets on the cluster and so on. But at least this solves the issue of not being able to commit your Git repo and so on. So at least until it reaches the cluster, everything is encrypted. So this is simpler to set up and this will work fine for smaller teams and for smaller projects. for bigger teams and projects, a better solution is to use external secret operator, secrets operator, or ESO. So ESO lets you use external secret managers like Hershey Cobb Vault, Google or AWS secret managers, Azure so on. So it can keep secrets synced with the external manager and provide support for most of the well known external secret managers. It's a bit more tedious to set up than seal secrets, but it is worth for it, for the security and flexibility it offers. And this is also ideal for, bigger teams as, seal secret requires individual developers to use the CLI to, create the encrypted, seal secret and so on. and, yeah, and it's not that scalable, whereas, external secret operator, have more flexibility. Let's see how it works. you deploy the, external secrets operator on the cluster and, you, will generally create a secret store to define external secret managers, connection details, like how you can connect to this and so on, or what mechanism you will connect to the external secret store and so on. Stuff like that. you will, you can now create your secrets in, in the external service, like for example, Vault, for example. now you create an external secret resource, yaml file basically that defines what secrets to fetch from Vault. You won't define any sensitive data here, you'll only define which data to fetch from your external secret manager. this can, so this files can be committed to git as it does not have any sensitive information. So you can commit them freely in git repose. and when you deploy this to the cluster, the external secrets, so when you deploy this, the external secret operator will be watching for any external secrets on the cluster. And it will, once it gets that, it will use the secret source information and it will fetch the appropriate, sensitive data from your external, secret manager vault. And it will create a normal Kubernetes secret on the, cluster for the pods to access. again, you would have to define RBAC and, so on to secure this. Because in the end, both these options still use kubernetes secret and it has the same issues as kubernetes secret being, base64 encoded string. But at least this is quite secure on the, secure outside the cluster, like putting it in a git repository and so on. this is a good option for bigger teams, and it's easier to manage secrets in a vault, setup like an external secret manager, so it gives you much more flexibility. Of course, you would still, you have to use RBAC and all other best practices here, but it's a better option compared to sealed secrets or regular secrets. Another option is to use, secret Store CSI driver. It's a native Kubernetes driver and it lets you use, external secret managers, similar to ESO. so you can use Hashi wall, Google, AWS, secret managers keywords and so on. so it uses a platform authentication mechanism. it can also keep secret synced with external manager and supports rotation and so on. it has providers for most of the well known external secret managers and the CSI driver is the most secure solution, and also the, but also the most tedious to set up. let's see how it works. you deploy the, the driver and external secrets provider on the cluster. so generally, depending on the provider you use, you would have to, deploy the specific provider, driver for the specific provider as well on the cluster. this works at the APA server level. hence, this is not going to create any actual issues. Kubernetes secrets here. So this is a slightly more secure option. so you can create a secret provider class, CRD to define the details of the secret to be fetched from the secret manager. next you will create your normal, deployments and pod definitions, et cetera. And here you will define, you will, reference the SecretProvider class in the pods volume spec. when you deploy this, when you deploy this, the driver will watch, for this. And when it finds the SecretProvider class, it will fetch the actual secret from the external secret manager and it will mount that, into the pod tempfs volume. So only the particular pod, which references the secret provider class will have access to that particular secret in the pod. and this volume will be removed when the pod is scheduled for deletion. So this is nicer as nothing is exposed as, Kubernetes secrets. So that even people who have access to read KU Secrets, will not be able to, decode the sensitive data. So this is a much, secure option and ideal for, large projects, enterprises, and, very, security savvy, tips. if you're, If you are interested, you can also find a blog post where I, detail these options and, provide, examples and so on. I'll also, I also talk about all the pros and cons of these different options in more details here. So please do check it out. So next, best practice would be to keep Kubernetes version up to date. Of course, this should be obvious because like any other software, like any software, it is important to keep the Kubernetes version up to date as well. and like any other software, Kubernetes also has bugs and issues and from time to time, there might be high severity bug that calls for a CV. Hence, it is important to keep the Kubernetes version up to date for, the server and for the CLI client. you can check the Kubernetes security and disclosure information website to see if there are known security bugs for your Kubernetes version. And if you are using a managed pass, GKE or, AKS, it should be pretty easy to upgrade. And, for on prem installations, there are tools like COPS, KubeAdmin, and so on. So that makes it easy to upgrade, clusters. So use any of these available options to keep your cluster up to date. next is, restricting admin access. So kubelet is the primary node agent running on each node. And by default, a kubelet's HTTP endpoints are not secure. we need to restrict access to kubelet, the API, and, restrict SSH access. this could allow unintended access if it is not, secure. And hence, definitely should be, number one, on the priority. furthermore, when someone access to a Kubernetes cluster, they can access the k, Kubernetes API server and SSH. So this should not be allowed. all non admin users should not have API and SSH access. So all this should be locked up. to limit node access, to limit node access, cluster access should be limited as much as possible. So disable SSH access for non admin users, secure the API using OIDC and RBAC, have proper, role based access control, set up for AROIDC setup, as we saw earlier. So that only authenticated users with sufficient roles have access to the Kubernetes API. Next on our list would be to control traffic between pods and clusters. So generally, pods within the same cluster will be able to communicate with each other. And if you have multiple clusters in the same network, there may be traffic between them as well. do not leave this all open. in most cases, you don't want different clusters to be talking to each other. And in most cases, You don't want pods to be talking to each other unless you have a microservice setup. Even in that case, you would probably have, proper ways for the pods to communicate with each other like service registries and load balancers and so on. So you don't want them just directly talking to each other. so do not leave this all open as it could lead to a compromised cluster. and, when a particular pod, has a vulnerability, this, open communication could lead to a bigger attack surface. So it is important to, reduce the attack surface as much as possible in case of a breach. and it's important to isolate, as a attack surface as much as possible as well. lock all this stuff. So you can use Kubernetes security policies to control traffic between ports and clusters and you only allow necessary traffic. So again, follow the principle of least privileges, just allow exactly what is required. don't allow wider access. Next on the list would be to use namespace to isolate workloads. This is often overlooked. Do not run all your workloads in a single namespace. Isolating workloads in different namespaces based on business needs is more secure and easier to manage with RBAC so that you can control access to namespace based on requirements and not have everything in the open. This way you can fine tune RBAC even further to let users access only what they need to see. You can also use Kubernetes network policies to isolate traffic between namespaces where applicable. So use all these tools at your disposal and lock access between namespaces as well. Next will be to limit resource usage. So as with securing APIs and cluster itself, it is also essential to set resource limits on how much CPU, memory, and disk space can be used by resources and namespaces. this secures your cluster from denial of service attacks when a particular container uses up all the resources. You don't want a single container in your setup to use all the resources at once. This is often overlooked and it can easily happen if you have a rogue application, if you have a rogue library in a particular application, which might suddenly, you know, hog up all the resources. you can use resource quotas and limit ranges, to set limits at the namespace level. you can use requests and limits, to set resource limits at container level as well. So use all these and define, resource quotas and limits. Next on our list would be to use monitoring tools and to enable audit logging. it is extremely important to monitor and audit our clusters. this way, we may be able to catch a break in attempt before it happens. it can, it can help you to keep an eye on the traffic, detect unauthorized access attempts. So enable audit logging for the cluster and use monitoring tools to keep an eye on the network traffic. to and from within the cluster, the egress and ingress traffic. Monitoring can be done using open source tools like Prometheus, Grafana, or proprietary tools. So enable alerts for suspicious activities, alerts for access, unknown access attempts from unknown IPs and so on. So have an alert for all these general, suspicious activities and irregular activities. Now that we, saw what can be done for cluster security, we also have to consider the infrastructure best practices because keeping the infrastructure, secure is also part of securing a Kubernetes cluster. So first and foremost, ensure that all communication is done via TLS, Then protect ETCD with TLS, firewall and encryption and restrict access to, ETCD using strong credentials. Set up IAM access policies in a supported, PaaS environment like, EKS or GKE and AKS. Secure the Kubernetes control plane, rotate infrastructure credentials frequently, restrict cloud, metadata, API access when running in a pass like AWS, Azure, or GCP, as it could expose sensitive metadata. And in, often it is the case that these metadata APIs are not secured. So look into those. we also need to secure our containers, right? It's not just, enough to secure the cluster and the infrastructure. We also have to. go granular and secure the containers as well. And securing the containers are as important as securing the cluster itself. Let's look at some of the best practices in that area. Number one would be to not run containers as root. This is extremely overlooked. And it's very important, do not run containers as root as this would give the container unlimited access to the host. So and in case of a compromised container or, vulnerability in a particular container, this would give, the attacker a wider, attack surface and root access to the host. So do not let this happen. avoid container escape and privilege escalations. always run the containers using a non root user to limit access. Use chown user, when using docker copy commands, for example. Always set up a, least privilege user for the container and use that. another often overlooked aspect, is, Using minimal up to date official base images, right? So especially during development since it's not very convenient, mostly Often you will just go with a standard os image and just focus on the application But for production always consider starting with a slim base image and adding what exactly you require there instead of going for a fat, full fledged operating system image. use the minimal and up to date official images as I mentioned. Don't use generic images. Go for slim or light versions. Maybe even start with the bare bone version and add the required dependencies yourself. This will give you much more control. This will give you much more control, if you want to replace a particular dependency or if you want to upgrade a specific, library with vulnerability and so on. This will also make the overall image lightweight, gives you latest bug fixes and security patches and overall is much more secure. for example, use deterministic image tags. Don't go for generic ones. Use the exact specific version that you want. don't give, ranges, et cetera. Remove all unwanted dependencies, packages, and debugging tools from the images, to make the image lighter as well as to reduce the attack surface in case of a vulnerability. only install production dependencies, don't go for beta or, test dependencies and so on. use official verified images from popular software for popular software, prefer LTS version when available, use a trusted registry for non official images and always double check non official images, look into the images that you are using and verify the image publisher. these are all extremely important. Next would be to prevent loading unwanted kernel modules. This is similar to the previous one. If you're using a FAT, a full fledged operating system, it might be loading a lot of unwanted kernel modules that will unnecessarily use up your resources, reduce your performance, increase the image size, increase the attack surface, and so on. use only the kernel modules that you absolutely require for the application. This will reduce the attack surface and give better performance. this can be restricted using rules in the, Kubernetes blacklist conf, at the node level or by uninstalling unwanted modules from the node, directly. this can also be done by using a very bare minimum base image and controlling exactly what loads and, what dependencies and modules are there. Next would be to enable container image scanning in your CACD phase. If you haven't already add container image scanning into your build pipelines, you would thank me later. This is extremely important. this can be helpful to detect known vulnerabilities before they are exploited. So enable, container image scanning in your CICD, to catch vulnerabilities before they become an attack vector. use OSS tools like Clayer or Anchor or commercial tools like Snyk, it's up to you. But use one of these tools, to enable, container image scanning. And, have, have your CICD pipeline fail if, There is a vulnerability in your container. these are quite easy to set up and they support most CACD, solutions. So there is no excuse to not do this. Next would be to audit the images. It is extremely important to monitor and audit the container images periodically. check for security best practices. If possible, make this auditing part of your CI CD pipeline. use Docker Bench for security to audit your container images for security best practices. It's a great tool to ensure that custom images you build for production are following best practices. Finally, use port security policies for containers. So this will help reduce attack surface and prevent privilege escalation. you can use the pod security admission to limit the container's access to the host. This is the successor to pod security policies from version 1. 2 and onwards. This will help reduce attack surface and prevent privilege escalations in case the container image is compromised. And this is more common than you think. More than a million Docker images were found to be affected by malware at different times. this is nothing to be, nothing to, ignore. set this up. And, with that, we conclude all the security best practice tips. thank you for joining. Hope you, had, some useful information from here. Hope you follow some of these and secure your clusters, better. thank you. You can reach me on, BlueSky, Mastodon, and LinkedIn. Thank you for your time.