Hello. Thank you very much for joining this session about securing secrets in the Gitops era, which we're going to store right now, because you'll see that. Yeah, we always talk about Githubs, which is super nice. We talk about kubernetes, which is super nice. But how about the secrets? How we can manage the secrets under these environments? Because you know what, I know that there is the resource Kubernetes secret in Kubernetes. But let me tell you one thing, a Kubernetes secret is not a secret at all by default. So this is the reason for attending this talk to learn how to manage all these secrets. My name is Alexsoto. My tutor is Alexsoto. My email is altovy@redhat.com. Feel free to reach me there. And also, I'm the co author of these four books, which basically the more important for this topic is the secrets and Kubernetes secret book or money, which is this one. Okay, so let's start. First of all, a definition. What is a secret? A secret is a piece of information that is only known by one person or a few people and should not be told to others. Okay. Yeah, it's that the Cambridge dictionary definition. Basically, today we're going to talk about the secrets that needs to manage by our application. It's not the secrets of our users. So we are not talking about how to store the users passwords into a database. No, we're not going to do this. We are going to talk about the secrets that our applications requires to work. For example a username or a password, or for example a certificate, these kind of elements. So just one thing, all the things related to security, it's about layers. There is no 100% secure system. Anyone who tells you that it's 100% secure, it's not telling you the truth. So what you can do with security, and for example, in this case for secrets, is creating layers of protection. So if one layer is broken by an attacker, there is another layer protecting. And if this also is broken, we've got another layer. So it's about creating as much layers around our system as possible. So today we're going to talk about two different phases of the secret. The first one, which is the easy one, is when we create the secrets and we want to store it into git because we are using Githubs. Okay. If you're using Githubs, then you need to take care of how you store these secrets into git and then say that this is the easy part. And then we've got the complex part, the part that is like, okay, now I've got my secret in git. Now I apply this secret into my Kubernetes cluster. What's happened with this secret? How can I store it correctly into Kubernetes? Because at the end, the secret is stored inside kubernetes. So how can I store it correctly? And you'll see that that's more complicated. Okay, so first of all, what is DevOps and Githubs? Okay, DevOps is the key to meet the insurable demand for delivering quality applications rapidly. So basically, what is DevOps? It's that we are able to deliver our applications faster and it's like an infinite loop, which is we code, we test, we build, we deploy, we release, we parade and so on. And we are continually doing this kind of thing. But, okay, that's nice. But one of the problems that we might figure out most of the time is how we can deploy this application into production so we can adopt DevOps. That's fine. All shift lab, everyone working together. That's fine. Yes. Testers, developers, operations, everyone together, this DevOps figure, right? Okay, everyone together. Okay. Yes. But now we need to put this service, this application into production. And usually it happens something like this, right? We've got our application, which is, this is our application, okay? And all the developers here just using the application to go to production, we say, hey, yes, that's my application, let's go to production. But who is this guy? Yeah, probably it's the product owner, right? It's pushing back. No, we are not ready. We need more features before we go to production. Or maybe it's the tester. No, we have not tested correctly. The application. Who is this guy? Yeah, probably this guy is the architect of the application on top of the application. Yeah, I architected this super well. Right? No worries, it's going to work. Yeah. And this, yeah, probably is the DBA admin, right? Like he's always saying, or oh, maybe we should know, go to production yet. Yeah, but we need to go. Or, yeah, but there's the SQL indexes that I need to update. Oh, and the developers asked me to remove a column. Oh my. No. So probably, probably this is the TVA, right? And the problem, and this is repeated again and again and again, maybe once every month, maybe once every three months. Or you say, hey, Alex, we are delivering to production once every year. That's fine. But it's always a hard work. And the truth is that this should be something like this, right? Just having an application and all of us moving forward this to production, and this is exactly what we're trying to achieve with Gitops, githubs. We can think that it's like an implementation of DevOps, which basically everything is in git. Git is the single source of truth. We are not only having our source code into Git, but also the infrastructure that is required by our application, also the scripts or the manifests that we need to deploy our application. So everything is in git. Okay, then thinks of this, the infrastructure is as code. So when we only not work with our source come, but also with the manifest. In the case of Kubernetes Yaml files that we want to use to deploy this application, this service into production, and not only this, also the YAml files that we need to configure. For example, our Kubernetes cluster. So everything is as code. And one of the great advantages of this is that now we can use our Git commands to deploy our application, for example to production. We can use all these things that we already know, but Git for deploying to production or to deploy into staging. For example, one approach that we could have is having one branch name it staging, which basically is the manifests that are applied to the staging environment. And then we can have a branch being the production environment, which basically it's the manifest that we apply it to have the application running on production. Then how you move an application from a staging to production, basically sending a pull request from a staging branch to production branch. So we send a pull request, everyone can review that. The pull request is okay, that it will work in production and finally merge it. And when we merge it, then the new application will be deployed to production. So you can see clearly here that everything is code, everything goes to the git workflow. There is no ending console there, no minimal steps. Everything is automated. Basically when we adapt the Gitops application delivery model, we are saying that we've got two git repositories. Of course this is opinionated, I know, but my suggestion is that you have always two repositories. One, which is the source git repository. Basically it's the source code, okay, the source code that developers does here. Then we've got a config git repository. This other repository is the manifest that we need to deploy my service, my application to the cluster. So in the source git repository, usually we've got here the CI CD part, building the application, doing some kind of code analysis. Finally create a container image and store this container image, internal image, registry. This is the first part that we do with this GitHub's application delivery model. But the second part and which is important is the config repository. This is the manifest, this is the YAML files that we require for deploying this application. Usually in the case of kubernetes here we will have a deployment YAML file. Then when we create a new version here of the image, we will send a pull request to this git repository saying hey, check the deployment Yaml file, change this deployment YaML file and add the new tag. So just find the image section of the deployment YML file and just change the tag value from for example 10 to one. 10. Okay, what's going to happen then when we make this change? When we change this deployment yaml file? So we're changing the manifest that are responsible of deploying the application. What's happening? Okay, that we will have here some kind of monitor. Okay, in this case I'm just showing here Argo CD, but it could be flux CD as well or any other detops, let's say in this way technology. Okay, in this case we've got here argo CD which basically will detect the change that we did in here. So the Argo CD will detect, hey, we've got a deployment Yaml file that has changed, someone has changed the tag. So it will detect this change and will take action. Basically this action will be apply this new deployment. So it's going to deploy the new manifest. So after RBCD applies this change in production, we will have the new version. And finally we'll get back to monitor for any change. Okay, so this could be an example of a Githubs repository. You can see here there is a lot of YAML files, one that creates animal space, another one that creates a service, another one that creates a deployment. There is also here more here for creating a database, a postgres SQL database. Right? So we've got a to do application, as you can see by the name, right? And then this todo application requires a database. Yes. So we put here, this is how we deploy the database. This is how we deploy the application. Everything is here as code. But take a look at this. Oh yes, of course, my to do application requires a username and password to access this postgresql instance. So what you do, oh yes, of course, you create a Kubernetes secret, right? Because we already know Kubernetes and we know that we've got the concept of Kubernetes secret which is used for storing secrets. So let's create it. So you create a postsecret Yaml file. The problem is this one. The problem is that a Kubernetes secrets, as you can see here, is a Yaml file or a JSON file which has this data and then it has a pairs of key value key. For example, in this case it's username and the value and then password and the value. And you can see here, this is these characters here that would say, oh, this is the username and password. Yeah, it's weird but it could be. The problem is that this data, this content is stored in base 64. So a Kubernetes secrets is not created to be secret at all, right? Because base 64 is an encoding process and not an encryption process. So every time that we create a Kubernetes secret, what we are saving there is the base 64 of the secret. So anyone can just take this value, as you can see here, do basically four minutes minus the code and get the real value. So we want to use git apps, that's fine. Problem how we can store securely these secrets into git because now anyone with access to the GitHub repo will be able to get the secret because it's in base 64. And this is exactly what we are going to see now, how to protect these secrets in git. And let me tell you that this is the easiest part of the presentation. Okay. For this reason, it's the Barbie. So first project that I want to share with you, seal secrets. Okay, Seal Secrets is a project that uses a public key cryptography for encrypting the secrets. So these YAMl files before they are applied to the cluster. So basically as you can see here, it's a Kubernetes controller. So it's a piece of software running inside your Kubernetes cluster. Then how this works? Well, basically as we seen before, as a developers, we need to create a Kubernetes secret. So we create a Kubernetes secrets as we've done all the time. So just a secret. That's fine. Then you can see here that I've got the sealed secrets controller installed. Then I'm going to use this cube seal minus minus format Yaml because this is a YAml file and I want to introduce as a cube seal, the manifest, the secret without being encrypted. So that is the native secret. And then I want to have an output which is going to be the sealed secret. So this is going to be a new Yaml file where the data of the secret is using to be encrypted. So when I run this command, basically this command will connect to the Kubernetes cluster, it will connect to the sealed secrets controller and it will say hey, give me the public key to encrypt this data. Because remember that seals secrets has a public key inside but also private key used to decrypt. So it's going to return me the public key and say hey, this is the public key that you need to use to encrypt data. And then it will say okay, now I've got this yaml file, I've got also the public key. So I'm just going to combine both and create a new object, as you can see here, where it has a new field named encrypt data and the username and the password, now it's not encoded but it's encrypted. Okay, so now I've got a secret object that where nothing is encrypted and thanks of Kubeseal, I've got here a new manifest where the data is encrypted. After that point I can remove the original secret. The original secrets needs to be removed. Done. I use it for encrypting, I encrypted. Now I've got my secrets encrypted here in this object. These native Kubernetes secrets can be deleted. And this yaml file is the one that I can push to my git repository. Why? Because data is encrypted. So if you do not have the private key to decrypt this data, no worries, you can push it to git. No one will be able to check whether it's the real value of your secret after that. Okay, Argo CD will for example detect this change, detect this drift and it will decrypt this. But notice that this is happening inside the Kubernetes secret, sorry, the Kubernetes cluster, it will decrypt and finally it will materialize this encryption secret as a normal secret into Kubernetes. Okay, so seal secrets is used for storing secrets in git in the convenient way, in a secured way. But then when you apply these sealed secrets, these secured secrets into the cluster, we're going to decrypt it and create a native secret again. But, and I want to be clear about this, you are secure while you are into git, then into Kubernetes. All these other operations of decrypting happens inside the Kubernetes cluster. So developer will never know which is the real value. Then there is another project which is the external secrets project. External secrets project is more or less the same thing, more or less the same concept, which is a Kubernetes controller running. But it works slightly different because maybe you're saying, hey Alex, we are not using Kubernetes secrets at all. So we do not have this problem because we are using for example, hashicor vault for storing the secrets. So, okay, we are good. Yeah, that's true. But how you consume these secrets? Oh yeah, we consume them running some manual steps. Okay, yeah, you can do it, but you can automate it with external secrets. Basically external secrets, it's a controller that is running inside, sorry, inside Kubernetes cluster. This Kubernetes cluster might have inside or not any of these secrets management stores. In this case, for example, hash code vault. And basically in git, we are just storing the configuration, we are not storing the secret itself. So as you can see here, the first Yaml file that we're going to run here is configure external secrets controller and where the secrets are stored. You can see here, I say, hey, look, secret store, I'm just using vault for storing my secrets. And this is where the secrets are stored, in this location, in this path. Okay, so it's a Yaml file. You can see here that there is no secret at all. It's just a configuration. It's just a configuration of here is where are the secrets that I need for my application. And then we also need to create another Yaml file which basically says, look, external secrets. You have here a connection to this hashicorp vault. But the secret that I need is this one. You see here, I'm saying, hey, this is an external secret. And here it says the secret store ref, it says it's the ball back end. Remember that this is the object that we created here. Okay? So I'm saying you need to consume secrets from here. You set a refresh interval of 15 seconds. It means that every 15 seconds it will check if there is a new secret store there. And then finally here, I'm saying, look, the path is positive SQl secret and I want to create this kind of object. So basically when I apply this, I will get a Kubernetes secret created inside. But again, the secret value is never stored in git. In external secrets, we are just storing the configuration of where is the secret and then this project will automatically pull the secret and create a kubernetes secret, a native Kubernetes secrets. Okay, just in case that you are using Hashicorp vault, let me tell you that there is a hashicor vault operator that does exactly the same thing as external secrets, but also it implements an automatic rolling update. So when a secret changes automatically the operator knows which pods are consuming these secrets and starts a rolling update. So let me show you these two projects I've got here. My kubernetes cluster running. Okay, I can do Kubectl get secrets and you see that yeah, I've got some secrets because they're coming by default. But you see that there is not my secret at all. Okay, now I'm going to show you seal secrets how sealed secrets works. Here you see that I've got a standard Kubernetes secrets, okay, I can do this postgres secret and you can see that it's just a secret, it's a native kubernetes secret. If I apply this kubernetes secret, okay, I will have this Kubernetes secrets applied into the cluster. But notice that if I push this yaml file to git, then everyone who has access to git will know this remain password because this is not encrypted is encoded. So what I need to do, I need to do remember Cube seal F Cube seal. Okay, notice I say minus minus format and I'm just checking, I'm just getting the content of postgresgate secret which is this one. And I want an output which is going to be a secret but encrypted. So I can do this. And now if I do a but of Postgresecret, you see that now the object is a sealsecret, okay? And the use password and the username are encrypted. So at this point these two files are equivalent but this is encoded and this is encrypted but they refers to the same. Then what you can do is just, what you can do is just delete this file or simply not push to git. So the object that you will push to git will be this one because it's encrypted. Now let me show you what happened when I apply this file. If I do Kubectl apply minus f postgresal secret. Okay, remember that this seal secret is encrypted when I apply it. Basically I'm applying this object to the Kubernetes cluster. Then the sealed controller or the secret seal controller will take this manifest, it will decrypt because the seal secret controller has the private key to decrypt it and then it will apply it. So if I know I do Cubecarl get secrets. Cubecarl get secrets. Okay, you see here that there is a secret. Okay? And if I do Cubecarl get secret, this secret minus o yaml, you can see that it's in a standard secret. So it's a Kubernetes secret. You see that now it's can encrypted. If you see the content here, this is the content of the original Yaml file. This is the secret that I decrypted into Kubernetes controller. And you see that the data is exactly the same. And if you see the owner references, you see that it comes from a sealed secret. Okay? So of course, now what we've got is everything encrypted in git, but not inside kubernetes. Now I can do kifcatl delete minus f of the postgres sql sealed secret. And you've got this bidirectional thing. Now if I do Kubectl get secret, you see that the secret is not here anymore. So we've got this bidirectional, we've got a sealed secret object in git. I applied this to Kubernetes cluster. Kubernetes cluster. Transform this seal secret into a Kubernetes secret and apply it. And to remove it, we can do the inverse. And it works. So he knows exactly, kubernetes knows exactly that this secret is a seal secret. And when I delete a seal secret, also needs to delete the native secret. Okay, I know that we are mixing with all these, with the things saying that hey, there is a secret, seal secret and so on, but I know that you get the point. Okay, now let me show you another example. In this case, external secrets. I've got here, I've got vault running. That's my instance of vault. And inside vault, I don't know if maybe locks. Let me check in vault locks now. Okay, well, I cannot show you now, but before the demo, I just created a new entry in vault. Okay, now let me go to here. This is the Yamls files. Okay, I'm going to show you first of all this cluster secret store where basically here I am configuring the connection to my vault instance. Just this, okay, I'm just configuring the external secret controller that, hey, there is a vault that I'm using to store secrets. So just keep in mind that I would like that you consume secrets from there. That's fine. Nothing is compromised. Okay, now let me show you the other file which is external secrets. Basically here I'm saying, look, I want you to use this connection, then you'll see that in vault there is a path with secrets foo. So you know that involved at the end, it's like a key value. There's a lot of more things, okay, but just to simplify it, it's a key value where we are storing in a place secrets in the correct way, right? Which are encrypted, protected and so on. And you set a key, which in this case is secret foo. And then inside this key, we can store several properties. And one of these property is named my value doesn't matter exactly everything. But what I want you to notice here is that the secret itself is not pushed into git. In this yaml file, there is no secret. It says where is the secret, but not what is the value of the secret. So you can clearly push these Yaml files to git. And then what's happened? One, when argo CD, for example, detects that there is this file or a change of this file, it will apply this, basically applying this, it will connect to the external secret controller. It will connect there and say, hey, external secret controller, go to the vault instance, take this value, my value from this key, and create a new secret. Name it example sync. So now if I do Kifcarl get secret, you see here that I've got example sync, which is a secret that I created 26 seconds ago. So it's this one. And if I do give catal get secret minus o yaml. Okay, you can see that here, it's a secrets. Well, there is a lot of things, okay? But here you can see that I've got the secret that I store inside vault, of course, in base 64. Okay, so basically with these two examples that I'll show you now, with steel secrets, you are encrypting the secret and you can store it safely into git. In the case of external secrets, you are not encrypting nothing. The external secrets project assumes that the secret is in a place in this case, for example, hashicor Vault. And the only thing that you configures is where is the secret? And then automatically, when you apply where is the secret, it will create a native Kubernetes secret into the cluster, as you have seen here. Okay, so think that it's more or less clear, right? Basically, it's how to solve the problem of storing secrets in git. So, no, we are stored in a correct way. These two projects, depending on what you're doing, if you are using Kubernetes secrets, then seals secrets is the best way. If you are already using a secret management system, then external secrets is the project. But all this time I'm just saying. Hey, okay, we are good, right? Because now we are storing the secrets correctly encrypted into Kubernetes cluster. You've seen that I do Kubectarl get secrets and you see that I'm creating the secret. And in git they are never exposed. But what's happened about the secrets that are then stored into kubernetes? What's happened with them? Because they're there. Sorry. Let me tell you. The first secret ATCD is not encrypted. What does it mean? It means that ATCD is the database where you are storing the secrets into kubernetes. So when we do a kubectl apply of a secret or a seal secret or wherever or the controller creates a secret into the cluster. Cluster or Kubernetes cluster stores this secret into ATCD. ATCD is like a database where you can consume or you or Kubernetes cluster consumes the element. And okay, if ATCD is not encrypted means that any secret that you store there, if anyone has access to the ATCD, it will be able to get the secret. Or if he has access to the ATCD logs, it might have the secret. Maybe not. Maybe yes. Depending on how you implement the logs, if anyone has access to the ATCD backups, then he can get the access. Right. Because ATCD is a database with passwords in plain text there. So yeah, this was detected by people from Kubernetes and said oh yes, yeah, we see that this is not the best way. So we're going to create a new object which is an encryption configuration object that we're going to apply it. So when we go here and you apply a secret. So when you create a secret into the cluster, it will go to a phase of encryption configuration which basically will encrypt the secret and then store into ATCD encrypted. Okay, this is how it works. So now you will say, oh, we are safe, right? Because I've got a secret in git which is encrypted. Then the controller will create a Kubernetes secret. But then we will enable this encryption configuration. So the secret will be stored in ATCD encrypted, right. Well this is not entirely true because there is a problem. ATCD is where we store the encrypted data, the encrypted secret. But also in the ATCD we're storing the keys for decrypting the secret. So what happened if an attacker get access to ATCD basically, it will get ATCD, it will get the secret. It will get the private key to decrypt the secrets. So everything is there. I can decrypt the secret. Thank you very much. Right? Yes. You thought that it was everything encrypted, but it was not. Well, it was encrypted, but the private key was also there. So it's like not encrypted. If you are a serious hacker, you'll get it really quickly, right? To decrypt this data. So possible solutions. First of all, move ATCD outside of the cluster. This is something that people really don't know, is that you can move ATCD outside of the cluster. And what happened that if ATCD is outside of the cluster, means that you can start storing things in different places. So you can have one ETCD where you store the secret, another ETCD that where you store the private key. It's super complicated. No one use it. But I just mentioned it here because the solution that I recommend you is this one using the KMS plugin. Okay, what does it mean, this KMS plugin? It works in similar way as this, as the encryption configuration. But instead of encrypting the content inside the cluster, it relays on a third party system. So in this case, I've got a secret. So someone applied a secret into the cluster. Then automatically the cluster will take. This secret goes to the third party that you configure it, and the KMS plugin section, in this case, for example, Hashicor vault, it will go the secret. It will do here, to hashicor vault. Hashicor vault will encrypt the secret, and then it will store this secret into ATCD. Okay? Then when we want to apply this secret, when we need to consume the secret, it will be the reverse order. We're going to the ATCD, we're going to the ball ball will decrypt the data and just then give us the secret. Much better, right? Because now an attacker might need to compromise two systems, which is more difficult, right? It needs to compromise ATCD to get the secret. And then you need to compromise the vault, the cache vault for getting the private key and then decrypt it. So this is more a convenient way, more secured way when we are dealing with passwords or secrets. Sorry, but. Oh, yes, that's much more. Right? And probably you already know this, basically what's happened when you are encrypting ATCD. But then you need to consume these secrets. Okay. That we can inject these secrets into the pod, right? Because at the end our application is going to run into a pod and we need to inject this pod. Sorry, we need to run this pod and this pod requires a secret, so we need to inject this secret into the pod. ATC is encrypted, but when someone, for example, a pod request the secret, the secrets is decrypted. Basically. What does it mean is that if you're using an ambient variables, you're going to inject this secret as an embarrassment variable. So not encrypted. Anyone could just go there and say export and just get the secret. Or maybe sometimes in the monitoring tools you are printing, right? Or you're monitoring also the MBM variables. So, okay, you will have in your monitoring system the passwords or in the locks. Some systems locks them on variables. So you will get the password into the locks. Another option, it will be as a volume, it's much better because as a volume it means that you are mounting the secret into a file. In fact, it's an in memory file system. Okay, so you are materializing the secret as a file. So it's not an MBL member variable, it's much better. But in any case it's super easy to get them because you only need to do an FS tab. And when you do an FS tab, what you're getting there is the location of this in memory file system. So you go to the location of the memory file system and you can get the passwords. Okay, the third step, the most complicated one, is using the Kubernetes API. You know that from a pod you can consume elements from the Kubernetes API. So for example, I can say I'm a pod and instead of injecting the secret, what I could do is say, okay, I'm just going to create a connection, a rest connection to Kubernetes API, for example, and consume from there the secret. So I am not injecting the volume, sorry, I'm not injecting the secret as a volume or as an immune variable, but I'm consuming directly from the Kubernetes API, which is more and more and more secure. That's for true, right, because the secret goes from ATC encrypted to the memory space of the application, unencrypted. But you will need to write a lot, a lot of yaml files. So I have an idea basically, and this is what I suggest all the time, is using hashicor vault. If you're using hashicor vault or any secrets management system, you are consuming the secrets directly. So no need for creating kubernetes secrets Yaml files no need for configuring YAML files, no need for injecting secrets as a file, as an MBM variable, which are not the most secure way. The most secure way is doing a direct connection to the secret management system using for example, HTTPs. You go there, you consume the secret and you store the secret in the memory space of the pod. So the secret is always encrypted here, then it's decrypted and moved using HTTPs, so it's inarrowly encrypted. And then here the HTTPs protocol will decrypt and put this secret in the memory space of the application. So if an attacker get access to your pod, we'll see no MBM variables with secrets, no files with secrets. Of course he could dump the memory, right? But yeah, probably it's super impractical, right, to get the secret by inspecting the memory dump. Of course it has good and bad things. The bad thing, I would say that it's a new tool that you need to operate. So if you are not using hashicle vault, you need to learn hashicore vault. If you're not using any secret management system like having Amazon or Google or Azure, you need to learn it, you need to operate it, you need to understand how it works, you need to configure it correctly, securely and so on. Also it requires code changes, so your application needs to change. For example as an mgomon variable approach, probably your application doesn't need to change, but in this case you are doing. A direct call from your application to the secret managers store means that of course you need to change your code to consume this secret. But I will say that they are the best. Okay, these tools, especially Hashicorp vault, are the best tools for keeping your secrets secure. It also implements monitoring for attacks. It also lets you audit what's happening with your secrets. It gives you a lot of functionalities for secret rotations. And one of the best things that it has is the dynamic secrets. Because you know what the best secret is? Having no secret. Because if there is no secret, then there is nothing to hide. And this is exactly what dynamic secrets does. And Hashicor Vault does extremely good job. Basically a dynamic secret is that you configure vault to inject, for example, the password to access to the postgreSql database. It injected automatically to the postgresql instance. So the administrator of PostgreSQl will never know the password because the password is generated and injected automatically by hashicor Vault. What's happened? If you for example, put a rotation of 60, 60 minutes, then Hashicor vault will update the password of the PostgreSQL every 60 minutes. But it's even more interesting because remember that this password is inside hashicor vault means that our application, the application that needs to connect to this postgres SQL database, it will get the password directly from Hashicor vault, the same password, and then it will connect to that database. So you can see that no one knows the password, only Hashicor vault. And Hashicor vault will connect to the postgres SQl instance, to my application instance, it will inject this password. And if every 60 minutes we change this password, Hashicor vault will inject this new password to both instances, which means that we as operators will never know the password. So we do not need to have any concerns about YaMl files, about any developer publishing the secret. No, because the password is automatically, dynamically created. Let me show you some example about this. I'm going here. Okay, let me go for example, just to have something quickly, I will share with you the details of this rip. Okay, now let me create for example, a secret. Create secret. Okay, this is my secret. I've got my Kubernetes cluster encrypting ATCD content. So now my secret, this is my secret here. I just created this secret directly, but it could be a sealed secret. Okay, but just for the sake of simplicity, I just created a secret. Now this secret has been stored into the TCD encrypted. Yes, but now what happened when I want to use, I think that I got here. Yes, I need to inject it in some way into my application. And in this case I'm using a volume, not an MM variable. I'm just setting a volume. You can see here, I'm saying, hey, look, I want to create a new volume which is named my secret volume. And the mount path is my stuff, secret of stuff. And here I'm saying that in volume section and putting this name my secret volume, which is this one. Okay, I'm saying I want to create the secret from my secret. You see that the secret that I created here, it's my secret. Okay, so basically here I'm just saying, but into this directory, the content of my secret. Now I'm going to deploy it deployment sQl. I always say that if I get it €1 for every time that I wrote incorrectly. Chief cutout, probably now I will be retired. Okay, now I've got here my pot. Okay, it's just starting, wait a bit. Okay, now it's running. Now if I do cubecatal exact minus ti of this, basically I'm simulating that I'm an attacker, that I get access to the pod. I can go to my staff and see the secret stuff. And here you can see that there is two files, one that is named username, another one that is named password. And basically the name of the password is the key and the content is the value. So you can go here, I can do a cat of password and you can see here that I've got my password, okay, which if you remember it was the value, my password. So that's what I meant before that. You can store correctly your secret into git, you can store them correctly into kubernetes, cluster into etCD. But when you inject it into a pod, you see that it's in clear text and that's what I meant. Then use hashicor vault and use the dynamic secrets because the best secret is having no secret. And this is what I mean that I'm not running, I just here have the code because basically if I run it, you will see nothing. But what I want you to see here. Here is where I configure, you see here, this is where I configure the postgresql connection. Okay? You see that they say, okay, I want to connect to a postgres. The username is admin. Okay, this is for the sake of simplicity, nothing else. But you see here, credentials provider Db. So I'm just saying, okay, no, there is no password here. You see them say quarkus vault. I'm just saying, hey, just keep in mind that the password is not settled by the user. It's not set it by an environment variable, it's not set it by using a volume, it's here inside vault. So basically what I'm doing here is saying the password, it will never be provided by any human, it will be provided by Vault. So vault will create a password in the positive SQL database, vault will create a password, will inject the password or this application will consume the password from vault. Okay. And then that's the password. You see a password? No, there is no password. Any problem with Yaml files? No, because we do not need a Kubernetes secret yaml file because it's already done by vault automatically. And any problem on injecting the secrets? No, of course, as I said before, you need to change a bit the code to make it run. Then just let's wind down. Okay, basically we are covering three phases, git in ATCD and storing the secrets in runtime, all of them. I mean maybe you might use seal secrets, maybe external secrets. That's for the git for ATCD. We saw the key MS plugin, so encrypt your data outside. And then we've got the problem of how to inject the secrets into the memory space or as a file, as an emul variable, I recommend you memory space. So again, just to finish, there is no good or bad thing. I just explained the problems that you can find and possible solutions. If you want to inject secrets as an MBLM variable. If it worked this way all your time and you have never had any problem, then go for it. But if now you say oh yeah, it's a good thing maybe that we should encrypt things, then let's move to for example, files. Maybe you've been storing Kubernetes secrets with ATCD unencrypted and it was fine because you protected really well ATCD and so on, then go for it, it's fine. But if you are, as I said at the beginning, if you want to add another layer of protection, because notice that you can protect ATCD, but if this layer of protection is broken, then the passwords are in plain text. So maybe you want to add another layer. So let's add a new layer, right? A new layer that is like, okay, maybe someone get access to ATCD, then it will find the secrets encrypted and so on. Okay, what is really important as well make secret key rotation. One of the key points to maintain things safe is rotating the secrets. Rotating the keys. Tools like falcos tag rocks and sneak helps you a lot of preventing and detecting these threats. Remember, no secrets is the best secrets. I said that before. And you've seen, right. If you do not have any secret, then no worries of managing them. And keep in mind that there is not a one or there is no black or white, so there is a gray, right. And probably you might need to coexist both options. Secret management tools like hashicor vault and also native Kubernetes secrets. So for this thing I think that it's important to understand secrets management systems, but also kubernetes secrets. And that's all. If you want to learn more about it, you can just scan this QR code and this QR code you will be able to download it. This book for free. This not, but this, yes. And yeah, we're covering a bit of githubs and also how to secure the part of the Githubs links about. Yeah, here, this is the link of the presentation. This is the link of the code. Here are some links and that's all. Thanks you very much for coming to this session. Have you enjoyed, have you find it useful? And if there is any question, remember that you can find me in email or Twitter. Thank you very much.