Hi all and welcome to this presentation regarding managing secrets with confidence. A comprehensive guide to using Hashicorp vault in Kubernetes clusters let me introduce myself. I'm Alain Lompo. I'm a senior software developer. I'm a DevOps and I'm security consultant in the public sector. I've been working with software development technologies for 20 years now. I've been doing also training and consulting. Let's see our agenda. First we'll be talking about handling sensitive resources in Kubernetes. Then I will introduce briefly Hashicorp vault and after that we'll deep dive into its architecture before seeing how we can integrate vault with Kubernetes. And then we'll see some real world scenarios. After that I will do some demos and we'll wrap it up. First, let's talk about handling sensitive resources. When it comes to handling sensitive resources in Kubernetes clusters, there are a few questions we need to ask ourselves. First, what are we to store and then what tools are available to do that? How are the assets stored and how do the application access the stored assets? In terms of types of sensitive resources, we can talk about credentials such as password API keys, certificates such as TLS and GPG keys, or also application configurations such as runtime arguments. Of course, depending on our application, there may be many more various data that are considered sensitive in our particular context. Regarding storing sensitive information, we should ask ourselves what are the available secret management storage solutions? How is sensitive asset intended to be used, what application framework is being used and what are the options for injecting external configurations? When it comes to kubernetes, the natural default is to think of using Kubernetes secrets resource since it's included in every Kubernetes distribution, but it provide only some form of protection since the values are not encrypted, but instead they are simply base 64 encoded, which means that anybody that can access them can also decode them. To create a Kubernetes secret resource in a Kubernetes cluster I can use a manifest like the following and use a tool like Kubectl to create it into my cluster. Now let's introduce Hashicorp Vault. Hashicorp Vault is one of the solutions out there for managing sensitive information. It's one of my favorite for various reasons. First, Hashicorp Vault is a very popular open source platform that helps us manage secret it helps us store these secrets, centralize its access control, as well as ensure compliance with various other security standards. In terms of its high level features, I can say that it simplifies secret management. It helps protect sensitive data. We can integrate it with various cloud and infrastructure platforms. It provides also a unified and secure solution for secret management. Now let's see. Hashicorp Vault architecture this present slide shows the main component of Hashicorp vault, and we'll first talk about the encryption layer called the barrier. When Hashicorp vault start, it tries to write to the storage backend, but since the storage backend is outside the barrier, it is considered untrusted. Therefore, the data must first be encrypted before being written to the storage backend, so that even if the storage backend is compromised, the hacker may not be able to use these compromised data, since they will remain encrypted until vault itself decrypt them at the start. Vault find itself in a sealed set and it must be unsealed before any operation can be performed on it. Unsealing vault is done by providing the unsealed keys. At initialization, Vault generates an encryption key. It's used to protect all vault data, and the encryption key itself is protected by a root key that is stored alongside all the other vault data. But it's encrypted itself by another mechanism called the unsealed key. This unsealed key is handled using Xiaomi's secret sharing algorithm, in which the unsealed key is splitted in a certain number of shards, among which there's a required amount of this shard that are needed in order to reconstruct the unsealed key and use it to decrypt the root key. Once that is done and vaults retrieve the encryption key, it secrets the data in the storage backend. It enters then in an unsilled state, and once it's in the unsealed state, it can load the configured audit device authentication method and secret engines. Let's see then how to secure the audit device. The authentication method, and the secret engines. They are all sensitive components. They are stored in vaults, and that means they are protected by ACL, and any change to them will be tracked by audit logs, and only users with permission can modify them. Now, let's talk about the role of the core component of hashicorp vaults. Among others, it process the traffic between the API and the other component of the system, but it's also enforced access control list, and it ensures audit logging is done. In terms of authentication mechanism, vaults is very flexible and present various authentication mechanisms such as using login and password, using GitHub, using app role, using LDAP, et cetera. Vault use policies in order to grant access to protected sensitive information, and these policies actually are simply named access control list rules. For example, we have the root policy, that's a built in policy, and it permit access to all resources. We can create any number of policies, and it allows us a fine grained control over the various paths that we are trying to protect. Vault operates also in an allowed access mode, which means that any access to a specific path will be denied unless there's an explicit policy granting the required access. Client tokens are the default and preferred mechanism for authentication used by Vault, and once a request has been authenticated, client token will be generated and it will be associated with it. Also a list of policies, as well as the user identity, through which the user can then in his subsequent request, access the sensitive information he would like to retrieve. Therefore, the client token is used for making future requests in an approach similar to the authentication cookies used in web application. But client token may also have an associate lease, which means that they may need to be renewed periodically to avoid invalidation. Talking a bit about the request authorization flow, the authentication request will be made by providing the client token, and the client token will be used to verify that client. That way, it can be ensured that the request is authorized and the associate policies will be used to authorize the client request. Now let's talk about the secret engine. The secret engine is a very flexible component of hashicorp vault. It's used to store and to encrypt sensitive data. In terms of its function. It receives a set of data, it will process them, and it will return a certain result. It can cover various tasks, but at its core it will be used to return the secrets. And once it returns a secret, the core will register it with the expiration manager. It may attach a lease id to it, and the client will use this lease id to renew or revoke their secrets. If the client ever allowed that lease to expire, the expiration manager will automatically revoke that secret. Let's see now. The audit broker and the audit devices. We talk about them in relationship with the core because the core will log the request and response to the audit broker, and thus they will be distributed to the various configured audit devices. In terms of other vaults activities outside of the request flow that we have seen, the core performs specific background activities such as list management, but it also handles specific partial failure case by using writehead logging with a rollback manager. But this is something completely transparent to the user. It's done within the core. Now let's talk about various aspects of integrating vault with Kubernetes and first, why is it important to consider that first, because containers are ephemeral, they are not the best place to store and manage their own secrets. Vault provide the possibility to centralize the secret management. It offers access control and auditing and it offers also integration with other tools. And there are various approaches to integrating vault, each one with their own benefits and limitation. Globally speaking, when I consider using vaults to secure sensitive information, I'm considering a use case where a given application can provide access to sensitive information to its users. But first, this sensitive information should be stored by the security team or other privileged user in Hashicorp vaults and then when the user make a request to the sensitive information, his credentials, his claims might be verified, the application will access Hashicorp vault and request to receive the secrets presenting the claims of the credentials of this user. Once they are verified, the application will be granted the access and the user can access the sensitive information he was looking for. In terms of integration of Hashicorp vault with Kubernetes Hashicorp Vault can also function as a KMS key management server through the help of an adapter that presents itself as a Hashicorp vaults KMS plugin for Kubernetes. Other than that, one of the common ways to using Hashicorp vault to inject secret into pod is through the Hashicorp vault agent injector that will be used to mutate an existing Kubernetes pod and inject into it a sidecar container that will be responsible for handling access to the secrets from Hashicorp fault. This is very interesting because this allows for example a legacy application to be able to have its secret management handled by Hashicorp vault even without being aware of Hashicorp vault itself. Now let's talk a bit about Kubernetes authentication backend. They are used to enable pod and service to authenticate with vault and so to be able to obtain secret dynamically after their authorization have been verified and in this case a token will be returned upon success. This following slide presents a bit what type of command I could use as security manager to enable authentication Kubernetes authentication with Hashicorp vaults as well as to configure its endpoint. Among the benefit of using Kubernetes authentication backend, there will be no more need for manual token distribution among ports as well as manual token management. I could access a tight control of access based on ports, identity and permission and I have a seamless integration with Kubernetes. It is also possible to confidence automatic token renewal and I can manage my secrets in a secure and scalable fashion in a Kubernetes environment. Now let's talk about using custom resource definition, which is one of the important aspect. Also related to integrating Hashicorp vault with Kubernetes. Custom resource definition are a way to specify additional resources into the Kubernetes API so that I can use these additional resources along with classic resource used in standard Kubernetes API. Not only can I create new custom resources, but I can also modify them and in this case through the operator it will detect the change event and will adjust the current state. Otherwise in case of conflicts it will launch a reconciliation process. This is an example of custom resource definition that may look a bit more complicated, but based on simple things like managing my server and archiving these actions, the benefits of using custom resource definition is simply that they are native Kubernetes integration. They are configured declaratively, they offer scalability as well as extensibility. All right, now let's see a bit of world scenarios that we could use with Hashicorp vaults. We actually have a lot of possible scenarios. I just selected a few that are relevant to what I'm doing currently. One of them will be storing application secrets. Another could be integrating with Kubernetes service accounts so as to grant this service account dynamic configured access to sensitive resources. Another scenario would be centralizing secret management through a centralized secret store that could be used with various teams without any interference and without any risk of secrets leaking. Another option will be to implement role based access control for Kubernetes secrets. Other than that, I can provide secrets to legacy app using Hashicorp vault and vault agent injector approach with sidecar containers. Or I could use Hashicorp vaults to manage SSI key for Kubernetes clusters as well as integrating with other security and compliance tools. Now let's see a simple demo of injecting secrets into a pod with hashicorp vault. So because the steps of my demos are a bit extensive, I have simply created slide of the demos instead of doing everything manually. So the first step of this demo will be to enable a secret engine on my Hashicorp vault server. I could do it with the following command vault secrets enable and I give a pass. Everything is a pass with hashicorp vaults. In this case my pass is internal and the secret engine is Kvv two. And in case of success I have a message that looks like the following. Next I would create a secrets at a specific path and here I can see that I'm using vault kv put and I give the path to my secret and since it's a key value secret engine I can then give the various keys and the values. In this case the first key is username, the value db read only username. The second key is password and the value is db secret password once I have created the secret I can verify that the secret has been effectively created using vault kv get and I give the pod and it will give me various information but at the end I will see the state of my keys and values and I could verify that both the username and the password has been effectively created to confidence kubernetes authentication I have similar commands vault enable kubernetes to enable using Kubernetes authentication and then I could use vault write give it part to the configuration where I could configure the Kubernetes host. Once I've done that I need to create a policy that will grant certain number of capabilities. In this case I just need read capabilities to be able to read the secret at the specified part so it will look like what we see in this slide. Vault policy write then I specify the name of the policy that I want to write inside I have which pad should be the target of this policy and which capabilities I'm giving. In this case only the read capability. After I've done that I will create a Kubernetes authentication role and I define the role using odd Kubernetes role internal app pad and what's important in the role is as we can see what I have highlighted here to which service account it is bound. The service account is called internal app and to which namespace we'll be using the default namespace and then which policies are associated with this role. The internal policy that we created. Internal app policy that we created before. Now what I need to do is to create the corresponding service account defined here as internal app. So I could use Kubectl to create that service account and once this is done I could test it on a simple application like what we see right here, my demo app. And what we could notice with this demo app is I'm using a service account name the service account given before internal app. So I can see now that I have created the demo app, its container is running alongside the other vault containers, the vault server here with name vault zero and the agent injector. Now let's inject secrets into the pod. To do that I will patch my previous deployment using this new manifest. And all we can see here is that by in dispatching I'm simply adding a bunch of annotations that are all prefixed with hashicorp.com and then I have various properties associated with this prefix agent injects through which role I'm using internal app and then I give the part to the secret. I use Kubectl patch to patch my previous deployment with this new manifest and then I can verify that the secret can now be accessed. All right, last but not least, Hashicorp vault can also be used in the context of CI CD pipelines. And as you can see here, it's used in combination with various other tools such as GiT, as well as sealed secrets and Argo CD that are used generally in the GitHub's approach or in CI CD pipelines. In this case, all we are trying to do is use seal secrets tool to encrypt the secrets before storing them in my repositories. And once these changes are done to the repositories, Argo CD will be able to detect the change and trigger the pipelines. And we can see that one step here is doing is it applies and decrypt the secrets. And then the secrets will be stored in vault and then they will be consumed over half the tp and they will be injected with CSI driver into the port so that at the end the port can receive the secrets in an automated fashion to wrap it up. I would say Hashicorp vaults is really a very interesting secret management platform for handling sensitive secret for kubernetes, not only for Kubernetes but as well as for other platform. It integrates very well with various platforms, it integrates very well with kubernetes. It allows us to perform secret management for various scenarios, not only what we have seen in this presentation, but many more use cases such as database credentials or certificate management and many more. So I will simply say I really enjoy using this platform and I will encourage you to try it and then thank you very much.