Hello and welcome to this talk titled Kubernetes security jumpstart. Like a real jumpstart. The goal of this talk is to start our security journey in kubernetes. After this talk, hopefully you will know all the different aspects of kubernetes that you need to secure and what security really means in the scope of kubernetes or in general. This is not an in depth tutorial on how to secure your Kubernetes cluster. This is collection of bunch of different tools and techniques and ideas you have to be careful about when you're trying to secure your Kubernetes cluster. The slides to this talk are in this link at tiny cc k eight s security. If you need to follow along and if you want to find these slides, you can find it there. So I'll start with the question, are computers less secure than real life? By real life I mean you and I, our daily day to day life, when we think about computers, we think of a lot of different things. But what I'm trying to ask here is our computers less secure than real life? So let's take some example. Is this secure? This door with the door code on the top and on the handle? You could say this is secure because you don't know what this door is. Or how about this? Is this secure things? Turnstile is trying to stop people from going into this space, but as you can see, there's a wide gap next to it. How about this? This is a door that has a push to exit, but that you can kind of slide your hand through and kind of open the door from outside. Hidden camera. This camera is trying to monitor things, but it is blocked by a sign. Well, how about these locks? These are padlocks, standard issue masters padlock you can buy from any store and people use that to lock their gym doors or something, some small locker. So one on the left is a master lock, it's fairly common. One on the right is a padlock makes by squire. It's a british company that makes some of the strongest padlocks in the world. The one here, it's a one could lock. And when it was created, it was declared the strongest padlock in the world. So the year after that they decided, you know what, that wasn't strong enough. We're going to make a stronger version of that. We're going to make like a two could version of the same lock. So that year, this lock was the strongest lock of the year. Well, they thought, you know what, that's not good enough either. We're going to go higher. We're going to have to make it stronger. So they made a five pound version of the same lock, and that was, I mean, this is one of the strongest padlock you can probably buy if you're not custom making it. So what's the point of all this? What's the point of you showing you real life scenarios or actual logs or like, security things people use in day to day life? Security is a spectrum. Security is not a thing that, oh, it's something is secure or not secure. Things can be more secure or less secure. Never just secure. You can't have something that is just secure. Well, in software, you can probably have security in software that you didn't write. Only secure software is where you have no code. But the moment you write code, the moment you have infrastructure involved, you have a spectrum of security. Well, remember this lock, the five pound lock? That is probably the strongest lock that you can buy in the hand of lock picking lawyer. It's a YouTube channel that deals with, like, lock picking. It took him four minutes to dismantle the entire lock with some tools. So security is hard. It is a spectrum. It is hard to get right, because attacking computer is easier. We can attack a computer by hundreds of other computers at the same time. In real life, we have a lot of assumptions that you can make right. For example, most, many of us drive here. How many times when you are driving, you have to worry about this scenario happening. I'm going to assume and hope it's not. All of us have to worry about this situation happening day to day. But when you are a computer, software, everything is an adversary on the Internet. The moment your application touches the Internet, everything on the Internet is trying to attack it and exploit it. Trust no one. This is the agenda for the talk today. We're going to talk about security and us, which we just touched upon a little bit. We're going to talk about security in Kubernetes context. And finally we're going to get started with Kubernetes security. Things talk in no way, shape or form a full, complete tutorial on Kubernetes security. Security, as I said, is a large spectrum and a lot of things is involved. But hopefully this talk will give you ideas on things you need to worry about when you're building your Kubernetes cluster or maintaining your Kubernetes cluster. My name is Mophi. I'm a software engineer and a developer advocate at IBM. I mostly do container stuff, write go code and collect stickers. If you need to find me in the social media you can find me at Mufi codes in any of the social platforms up above. So let's started talking about Kubernetes security. Kubernetes security has few different aspects to it. Number one is application security. The application you run on Kubernetes next is container security, next image security, next infrastructure, then network security. And finally we're going to talk about runtime kubernetes security. Let's talk about application security a little bit. For application security use best practices for application security. And this may sound a little counterintuitive because I'm not really giving any proper advice because it is quite hard to give you advice or information about application security because it depends on the application you're writing, the runtime you're using, the language you're using and everything around your runtime. So just use the best practices for the language runtime and the environment you're using it for. As Gordon Ramsey could put it, I can't teach you that because that's common sense, although a lot of this common sense information are not common at all. But again, try to use the best practices. So let's talk about container security. No unknown base image. So between the two here, pop quiz, which one do you use? We have on one side we have movie code, Golang Super Secret 116 and Golang latest. Some of you may answer it's the right one because the left one is an unknown base image. You are right, but you're also wrong because none of things why? Because you have to lock down versions you can't guarantee. Just because the Golang latest is an official base image, you can't really guarantee this. Golang latest is the Golang you build your application on top of this latest image tag is a terrible image tag to use because that latest keeps changing with every new version that comes around. So if you build your application for Golang 112 and latest Golang 116, 117, 118, whatever comes out next might have some breaking changes that now breaks your application. So never use latest. So let's look at two Golang versions. So one of them is Golang 116 three Alpine 313 which is very specific. It's a tag that is pretty good. It's not necessarily the best, but it's still pretty good. But the best way to lock down version is by using digest Sha you have like the image itself and the Shia is built using the image content. So every time you use this image you are guaranteeing that you are building things image on the version that you tested and have verified. Your application works so use digest Shah when possible. And it's pretty much if you're using any kind of like image registry, it's almost always possible to use digest Sha. So make sure you use it. Next update for vulnerability remediation. So if you're using something like node and NPM packages, every once in a while you would see that your NPM packages have some vulnerabilities and you need to update your images. So there are tools we're going to talk about later that lets you kind of look at your images and find out if there's any vulnerability in them. Next is limited attack vector. What does this mean? So this basically means you should try to use smaller base images. So if you have a choice between Alpine and Debian, you should probably go with alpine because it is a smaller image with less things in it. So you have a less chance of exposing something that you didn't want to expose. And if you have a compiled language that can take binary, definitely look into scratch images. So scratch are empty container based images that only can run compiled binary. So if you're using languages like Go, C, C Plus plus or rust, or using Java with GralvM, you could take advantage of this container base image that has nothing other than the binary. So your attacker doesn't have anything to attack your container images on. So let's talk about image security a little bit. So for image security, the best practice here is least privileged access. If your user doesn't need to be root, make sure you're never root. Always use some user creation within the container image so that you're running as user 5000, which a random user who doesn't have any special permission in the container image. One thing I always like to say is root is the root of all evil. If your container image, if you're running as root, if somehow that image escapes now in the host, that user is now root as well and can wreak havoc. So you want to make sure your container image is running as non root users. It's kind of like things situation, right? Like your container is that small rhino inside the enclosure, but somehow it kept in gap. Now this rhino is out in the world, but now this rhino is also a root user. So outside and out in the world that this rhino can do anything they want. So you have to make sure the cage you're building for your container is really secure. Also, if it by mistake does go out, it doesn't have any permission to do anything extra. Let's talk a little bit about infrastructure security at this point. So again, Kubernetes is abstracting away a lot of the infrastructure, but underneath all of it, it's still, there are bare metals or vms or what have you. So we have to make sure that our underlying infrastructure is secure to make sure our application is secure. So don't expose underlying infrastructure. That kind of seems like common sense, but like in the last few years there have been a number of these exploits where hackers have attacked the master node or the worker node itself. So where Kubernetes security was not broken, but they broke into the underlying vms and master nodes and got access to it. Famously we had the cryptojacking of the Tesla Kubernetes servers where their master node, the Kubernetes dashboard, was actually insecure without any password. So they found the master node IP and just like accessed it directly from their computer and now then had AWS s three secrets exposed. Metadata is more valuable than you think. So when you're a Kubernetes cluster, information you get to yourself. You have information like your master node IP. When you're using Kubectl to connect to it. This information might not seem like without the secret it might not seem super valuable, but the moment someone knows your node IP, then they can started even start doing some DDoS attacks on the node. So if your node is too busy like just serving random requests, your Kubernetes cluster might go down because it can't reserve your network traffic hardened nodes. So you should make sure that your node doesn't have SSH enabled. That's just like security 101. At this point you could make sure that there is no other way to access your nodes. Don't expose the node IPS if you can, and all of these good practices you can start doing. So. Make sure your node underlying infrastructure itself is following the best security policies. Update regularly. So every once in a while under Linux like Ubuntu or like a red hat Enterprise Linux, whatever underlying node Linux environment you're using might have some vulnerability that leaks something or just something is vulnerable. So you have to make sure the underlying infrastructure is updating as well as update the Kubernetes version when the new version comes out after some testing. Because if you have a too old of a version, you might just open yourself up for some vulnerability that was fixed in a later patch. And just by not fixing the zero day vulnerabilities you're just opening yourself up for these kind of attacks. So let's talk a little bit about now. Network security. So Kubernetes is environments for microservice applications. So you have a lot of network traffic, both internally from service to service and externally from service to other applications. You have to be very careful about making sure your network security is up to date and working to keep everything secure. So network segmentation. So this is the idea of making sure internally, within the same Kubernetes cluster, you have segments where your application is isolated to talk to, only the application they need to talk to, rather than opening it up to everything in the cluster. So by default, any pod in any namespace in Kubernetes can talk to any other pod in any node. But that is not the best system for security. So what you want to do is make sure that you are locking it up so that it can only talk to things they have to talk to. For example, you can use network policy, and here we are only allowing incoming traffic from front end to go to back end and nowhere else. So that my back end pod selectors will only allow traffic to come in from front end. So if someone gets access to a random pod in your cluster and someone just hijacks that pod, they won't be able to make network traffic calls to your back end if this policy is in place. So some runtime Kubernetes security practices that you can use is use RBaC. RBAC stands for role based access control. So RBaC tries to answer this question, can subject verb object. One of the example this could be can user list pods. So every time you set up an RBaC rule, this is the rule you're trying to set. You could also set RBaC rules to individual namespaces. So then the question becomes, can user list pod in namespace? So you could have RBaC rules that are either specific to entire cluster or specific to single namespaces. So one of the examples, so this right here is example of a role that is bound to a namespace. So in this case, namespace default, we're creating a role that allows our user to list pods. So get watch and list pods. And we're doing a role binding to connect that role to a subject. So in this case, our subject is a user username is Jane, and role is then is going to be a read pod reader role. So things user Jane now will be able to get watch or list pod. So the command kubectl get pods will succeed. But if they try to do something like kubectural delete pod, that would fail because this user Jane does not have permission to delete pods. Define quotas. So every resource you create, you can define some sort of quota. In this example we're creating a resource quota and adding some sort of limit like how much memory you want to give, how much cpu you want to give, or in every single pod you can also create some sort of resource coder by saying we want to use 00:25 cpu, 128 megs of ram and also limit that kind of like different limits as well. So why is using resource code a security mechanism? So the ways resource coder help is limit attack surface and damage. So let's say one of your pod gets compromised and someone gets access to your pod if there is a limit, if the pod tries to access too much resource, kubernetes API can then kill that pod if it's asking for too much resource. If you don't have a resource limit, that pod in theory can just request too much resource and block out all other application that's running on your cluster. Also with resource coda. Another side effect or the main reason you want to use resource coda, that is you can get better resource utilization. If the scheduler knows upfront how much resource things applications is asking for, it can schedule it in a location that has exactly that much resource available and thus you can reach over like 70% to 80% cluster utilization. So that's why resource quotas are great. Finally, when you actually put a resource quota in your applications on your pod and deployment, Kubernetes actually marks these pods and application with a higher priority. So if you have limit and requests both set and are equal, your application is quality of service is rated as guaranteed. That means this is the last thing Kubernetes will delete and usually won't delete ever. And if you don't have any limit or request set, what happens is your quality of service is listed as best effort. So if something more priority comes along, Kubernetes API might actually delete this particular application. So putting resource quota is not only useful for security, it's also useful for your application so the application doesn't get booted when there is more resource needed. Monitor and log and this is not necessarily anything specific to Kubernetes. Anytime you have application and runtime, you want to make sure that you're monitoring and logging those information into some location, and so that if something does go wrong, you can immediately find it. And even if you immediately can find it, you can have a location where you can go find information later. Pods are ephemeral. So if you have pod logs just printed out on the pod itself, when the pod dies, something goes wrong. Those logs will be gone as well. So you'll have no way of knowing why a pod died. So I have some sort of central logging mechanism, hopefully outside the cluster that is constantly adding these logs into so that you can go back and trace why something went down. If some bad actor tried to make too many requests too quickly, you can then look at it and have some mechanism, have some way to alert or have some way to kill that thing. But you can't do that without having a location where you're collecting and aggregating this information. Admission controllers so admission controllers in Kubernetes is a way where every time you make a request or every time you try to create some new resource on Kubernetes, you can run those resources requests through this admission controller. So admission controller kind of looks like this. When you make an API request using Kubectl or directly using the Kubernetes API, it takes the authentication, all that is done. Then it goes to mutating admission and this webhook can mutate your request. And then after the request has been mutated you can then go through validation and validation can stop a deployment if it doesn't follow the validation webhook. And finally it gets persisted to etCD. And then at some point the reconciler takes that information and make your cluster state that. So why is admission webhooks useful? So you can use admission webhooks for pretty much anything. But one of the key security features they can do is look at some policy that you have in your company and make sure each of the resource that you are applying or deploying to your Kubernetes cluster follows those things. For example, if you want to make sure that you have quotas defined on all resources, you can put that in a webhook. And every time a new resource is being created, if it doesn't have a quota specified, either you can have a mutating webhook that adds a quota automatically, some default quota, or have a validating webhook. If it doesn't have a quota defined, it can just reject the application of that particular deployment. And it could be for a number of things. If you're trying to add privilege escalation, if you don't have AC Linux profile set and any number of things in between, you could do that using admission webhooks encrypt at rest. So by default the HCD database that where Kubernetes stores all its states is not encrypted. So to secure your information like secrets and other things, you need to make sure that your ETCD database is encrypted at rest so that no bad actor can access that information. Well, as Gandalf would say, keep it secret, keep it safe. So now let's talk tools. We talked about all the different things we need to worry about. Now let's see what are the tools in the ecosystem that we have to our disposal to make that job easier for us. Always be careful with your tools because it's very easy to burn ourselves when we are trying to use a new tool that we don't really take into consideration, what are the effects that could happen? But with that in mind, I'm going to talk about some of the different tools that touch upon different aspects of security. Most of them are open source. There are some paid products. None of them are endorsement. I have tried some of them out, I have found many of them in the ecosystem. So none of these listed here are like, and they are not listed in a particular order. So don't take this listing as any recommendation or kind of a vote of confidence. The ones I have tried and the ones I have kind of played around with, I can say if they are good and decent at what they do. But again, when you are trying to do that for your application, your company, please do some more research into them to find. But if they fit the need that you have. So let's first talk about image scanning and analysis. So when you are building container images or you have a container image, you want to make sure that this image doesn't have any vulnerability. There are a number of image scanning tools both in open source and paid. Some of them would be encore, Claire Dockle, Cubexray, sneak, trivi these are all tools. Some of them are free Apache MIT licenses and some of them has commercial offering tied to them. So they are part of a larger commercial offering. Let's look at a demo of a couple of these tools here in the list. Dockle good with tech test image v one. So guru tech is the creator of this tool dockle and they have a test image that you can run this against. So you're going to run that against this guru tech and this will just like download the image and scan the image, all the contents there and you'll see that we have some fatal things that are coming out of this. Talking about instead of copy we should use add do not store credential environments, clear app get caches. That is to make sure our image size is smaller. We're using avoid empty's password. There are a bunch of things that are bad and you can actually put this in between your CI when you're trying to build your image to make sure your image doesn't have any problems. So if you see any of the fatal errors you could stop your CI build and fail the CI build. Another image that does similar thing is called three v. With three v we can point it to one of our Golang latest image and it will look at the image and it will find bunch of things and many of these things are from the CVE. So these are list of container vulnerability that are found for Linux and it can list bunch of these things, right? And based on these issues either we can update our image or we can fail this build depending on if the error is something that we need to update. You can use both dockle and trivia in your CI and in their documentation. They have documentation for how to use them in your CI pipeline and make sure that you are taking into consideration that your image scan like whatever value you get from your image scan. So let's move on. So let's talk about runtime security a little bit. So for runtime security in Kubernetes what we are talking about is your cluster is ready, your cluster is running. We want to learn things about that cluster, whether or not the cluster is set up properly or the things you deploy to things cluster. So again these are a little bit more involved to set up. So we're not going to look at a demo, but there are some really big hitters here. For example Falco is a sysdig project. They do a lot of these security things that you inject into your Kubernetes cluster. They monitor the cluster real time and every time something bad is happening someone is trying to access the cluster in a way that is not allowed. It can create this monitoring alerts that you can look at. You can also set up SC, Linux and Seccomp to make sure that you are limiting the amount of things a container could do within the cluster. And these are definitely like each of them demand their own talk in their own rights. But you could set up these kind of rules to make sure that your container has the most least amount of access to do things in your cluster as possible. So let's look at the network security a little bit. So for network security there are a bunch of network overlays in Kubernetes and each of them aside from the network policy that is based on Kubernetes itself, if you are using any of these other things as your network overlay, for example in IBM cloud, Kubernetes we use Calico as a network overlay, and Calico has things you could do as well in terms of network policy that can go even more fine grained compared to the general Kubernetes network policy. So if you find out exactly which network overlay your Kubernetes cluster is using, you can make use of the particular network policy rules from your provider to make sure that your cluster network policy is really secure. Secrets management and this one is a tough one, because by default kubernetes secrets are not necessarily secret, they are just base 64 encoded. Although you could encrypt your HTTP cluster at best, even then your secret anyone who has access to the cluster can be read by and then converted base 64 from regular text. So creating secret in kubernetes is not probably the best idea. So the way you want to create secret is probably using some sort of secrets management, either using some kms from your cloud provider where your Kubernetes cluster is, or using tools like vault where you have either deployed yourself or managed from Hashicorp. Get vault that you can store secrets and inject secrets at runtime directly to your application for image distribution. There are a number of tools. You have grapheus in Toto Portieris and each of them does something a little bit different. But all of them has the idea that if your image, when you're distributing an image, how do you make sure that image is coming from the right place? Especially portiers has admission webhook that can look at an image and make sure this image is coming from a whitelisted source. So each of the tools, they are open source so you can deploy your own version of it. They're dealing with the idea how you can distribute an image from a secure source and securely to your end user. Security audit so for your kubernetes, there are ways to audit both the deployment yamls and the cluster itself. For example, tools like Kubebench or kubernetes, they're both from Aqua security. You can run it against a running cluster and that can check against some fixed rules to see if your cluster is set up properly. Kube audit is pretty cool. It can look at a YAML file that you have about to deploy to your cluster and check against the YAML file to see if the YAML file is following proper Kubernetes guidelines, and it can also fail your continuous delivery pipeline. If the YAML file you're about to deploy is not up to mark, you have Kubesec does something very similar. It can give you a scoring and it can fail your build. And finally, open policy agent in this scenario is probably one of the best tool that is out there for auditing your running Kubernetes cluster deployment where can define any kind of rule. So all the things we could do with admission webhook it is quite difficult to write your admission webhook from scratch with open policy agent you can define all that and more using very simple rules and logic. And every time you try to deploy something new it gets validated against the open policy agent webhook and you can then stop deployment because it doesn't match the set security rules that you have in your company. Let's look at a couple more demos that showcases a couple of the tools. We're not going to talk about open policy agent in this one in terms of demo because again that's a much more in depth demo and definitely deserves in its own talk to cover all the different things you could do with open policy agent. So we are running a Kube audit against the deployment that I have and this is checking for all the different things that we have. So if I run this against my deployment Yaml, you will see that we are getting a lot of errors. So one of them here is app armor settings is not set up, automount service account, token true and default service account. So in this deployment YAmL, we are not using like a special service account, we are using the default service account for the namespace which is not best security practices capability or security context missing. We are not adding any specific security context. We are not doing run as non root. So run as non root is not set to true allow privilege escalation neil so we are not also stopping our container to privilege escalate to root here either. So all these things we should set but we are not in this yAml. And so this cube audit tool is failing this particular YAML file. So if we set it up in our CD and run cube audit when you are about to deploy our YAML file, we could fail the build because our deployment YAML is not up to mark. We can also do something similar with Cubesec. Cubesec also can scan our YAML file and in this case we are looking at some, we get a score of four. So we got a passed because we have a limit set, we have a memory set, we have cpu set as well as request memory is also set. Some of that they also give some advice in terms of not directly errors, but give some advice like set app armor use service account, seccom root file system non root and run as user. So all these things, as you can see, both Cubesec and Cube audit does a very similar thing. So you can choose either of them in your pipeline to make sure that you are failing builds that doesn't follow the best security policies. Now finally, let's talk about some of the end to end commercial security products that are out there. So the ones I talked about, most of them are open source, but some of them are tied to some bigger commercial offering. With commercial offering we get enterprise support as well as probably a nice dashboard that can constantly be scanning your kubernetes cluster and updating you with the vulnerability that it found. So there are a number of them obviously, because again this industry have a lot of choices for almost everything. I list some of them here. This is in no way shape or form a complete list. So when you're doing your security you may come across something newer or something even better than this. But there are some of them I'm going to talk about. You definitely have aqua security. They put out a lot of open source tools in the community as well. There are a bunch of them in the list like sysdig as well, twist lock and a number of them. And there's like a quick comparison between some of them. And each of those tools have these different capabilities. And just because something has more capability, that doesn't necessarily mean they are better or worse than the other tools. It just means as of now and they are claiming to have all these different tools. It might be possible for you to look at these tools when you are trying to use one of them, that you may not need all kind of all of these capabilities. So for your application you might look at one of these tools and see that the capability they offer is all we actually need. So definitely keep this list in mind. Look at this and look at all the features each of them provide. But also keep in mind that just because having a huge list of different things some of these tools has to offer doesn't necessarily mean how good each of those tools are. So again, I would implore you to kind of do your own exploration, try them out, use the trial and see which application here fits your need the most. Finally, let's talk but service meshes. Service mesh, there are quite many of them now. The notable ones would be istio Linkerd one and two kuma mesh. So service mesh are a collection of best in class solutions that brings in bunch of things that right now is quite difficult to do just in kubernetes. And you have to do a lot of these things yourself to do them well. So service meshes control and monitor service traffic. They are invisible to dev team, which is a great thing. Your development teams won't have to worry about where your application runs to build really cool applications. And finally, they can enable mtls between service to service traffic. So when you have mutual TLS enabled between service, what you can achieve from it is if by any chance one of your pods gets compromised without the proper TLS certificate, that pod won't be able to talk to some other pod it is not allowed to talk to in the first place. So you can really limit the amount of things a bad actor can do. Even if they get access to your application? Even if they get access to the application pod. It also lets transparent monitoring of network traffic. So you'll automatically collect these metrics from the network traffic as they're passing through. Can kind of look at what exactly is happening in your cluster if one of their pod all of a sudden starts to make too many requests. You can monitor that, set up some alerts and take action based on that to make sure that your cluster is not going crazy. So let me ask this question again, right? We talked about a lot of these things and after talking about this for about the last 2025 minutes, are computers less secure than real life? Right? Like we have a lot of problems in computer security. At the same time, we have a lot more tools too to make sure that this security vulnerability doesn't affect us directly. As I said, and I'm going to say it over and over again, security is a spectrum. Nothing is like secure. It could be more secure or less secure. Never just secure. When I think of kubernetes security or security in general, this picture comes to my mind. Let's say our bike is the application and we want to make sure this application is secure. So all we are trying to do is put more and more and more locks on things bike, so that at the end of the day it's not worth for the thief to try to get rid of all these locks to steal our bike in terms our data and application. So when we try to secure our application, all we are trying to do is to make sure that sealing the data, stealing the information, breaking into this application is not worth the time. With that, go out there and secure your kubernetes and your applications in general. Hopefully I have managed to show you what are the different things you need to worry about when you're trying to secure your kubernetes cluster. And I'm trying to free your mind, but I can only show you the door. You are the one that has to walk through it. Great. DevOps Admin Morpheus said that to Neo once and I'm saying that to you now. With that. That is the end of the talk. Thank you so much for spending the last 30 odd minutes with me and listening to me about kubernetes security. I'd love to hear your stories, comments, questions and any cool security stories that you have. You can find me at movie codes and if you have any questions, feel free to drop by and ask me on Twitter. Thank you.