Welcome to this talk. Today I'll be talking to you about every developer's or administrators or operator's favorite tool, which is SSH, the same Ssh that allows us to remotely and securely connect to Linux securely connect to Linux remote Linux servers over an insecure network. Now, at the moment, the most popular way of authenticating SSH access is using SSH keys, and that's what I'm here to talk about. This talk is to show you or shine more light on why Ssh keys aren't so good, how they are doing harm to your infrastructure, and to convince you, or to show youll how SSH certificates are superior and why you should adopt them. Now, SsH keys are actually not bad. At least they are better than using passwords, plain passwords. Theyre they are convenient and easy to set up, and on their own they are actually quite secure and cannot easily be hacked, thanks to the mathematics of private keys and public keys, which I'll get into later. However, traditional SSH keys usage has private keys stored on digs, which means that they can often be stolen using malware or red if a team member's device is compromised and they have Ssh keys that aren't protected with passphrase on their device. Now, even when SSH keys stored on Dix are protected with encryption passwords, they can also be stolen using those passwords can also be hijacked using brute force attacks because people are susceptible to using insecure or easily guessed passwords. And when SSH keys are stolen, bad things happen. In recent times we've had several terrifying stories of companies that have suffered severe data and monetary loss due to one or two compromised SSH keys. We have companies like Godaddy, Shapeshift, Capital one, and just to name a few. Now, I won't go into the details of what happened with these companies because these stories are available online and you can read them on your own. The aim of this talk is to spark discussion and for us to examine really the inner workings of how traditional SSH key authentication works, and consider if there are better ways to manage SSH access in such a way that these incidents are less likely to happen, and in such a way that SSH access is safer. The question is, is there a better solution? And yes, there is, and that's SSH certificates, and that's what I'll be talking about in this talk to show you how SSH certificates are superior to using SSH keys and how you can start using them. But before I get into all that, I'll just take a moment to introduce myself and my company and then we'll proceed. Yep, that's me. My name is Linda Ikechukwu and I'm a developer advocate at Smallstep Labs. Now, small step builds Small Step is a company that builds toolkits that empowers developers, operators, DevOps, engineers, it administrators to be able to easily and seamlessly adopt PKI and use digital certificates, 509 certificates, SSH certificates to encrypt and secure all their infrastructure, from internal websites to kubernetes to docker containers to databases and what have you. So now in this talk, I'll be shining the light on how SSh keys work and reasons why they are bad for you. And then from that we'll move to looking at what SSH certificates are, how they work, and how they are superior to traditional SSH keys authentication, why they are great for you. And then after that, we'll look at how you can start using SSH certificates. What are some tools that are available out there that you can start using to deploy SSH certificates into your infrastructure? And then lastly, I also shared some links that will help you learn or set up SSH certificates in your own infrastructure. So let's get right into it. Why are SSh keys bad for you? Now, before I proceed, I'd like to state one thing. SSh keys are not so bad as I mentioned. In fact, I would say that when they are properly secured and managed, they could be the simplest solution and probably the ideal or preferred solution for a small of, say, ten to 20 devices or users or developers, or a small home lab. Because right now, with the addition of the SSh key types, a server can now enforce the use of hardware bound private keys for SSH. And this prevents sh private keys from being stolen from devices. And that's great. But as we proceed in this talk and look at the walkings of Ssh keys and how it works underneath, youll discover that SSH keys are just not scalable or robust for larger organizations who have fifty s or hundreds or thousands of servers and users. Do youll hand out dongles to every single person who needs Ssh access? That's a bit of an overhead, yeah. So let's take a look at how ssh key authentication works. Ssh key authentication works kudos to the magic of asymmetric cryptography, as I mentioned, which is the special correspondence between a public key and a private key. So if I send you a big random data and you sign it with your private key and send it back to me, if I'm able to use youll public key that I know to decrypt the signed random number that you've sent to me to obtain the original random number that I sent to you. Then I can be sure that I'm talking to you because you are the only person who has your private key and your public key and your corresponding public keys, which I have, is the only key that can decrypt the signed random message that you've sent to me that can correctly decrypt it. Yeah. So when you ssh into a server using your public key, the SsH server challenges your client to sign something with your private key, and when it does, that signature or that signed message is checked against the public key stored on the server. And if it matches, then you can successfully log in just as this image depicts. Now this may be an oversimplification, but it's more or less how SSH public keys authentication works. SSH certificate authentication works the same way, but with an important twist that we'll get to in a moment. Now how so we get the hang of how SSH authentication works. Now, when a developer, like I mentioned, to ssh into a host using a public key authentication, the host needs to know youll public key. Now, by default, youll public keys have to be added to the authorized key file of every host or server that youll like to have access to. So a normal SSh key onboarding or deployment involves a developer first generating their own key pair, which usually involves some baroque incantation of SSH keygen. Then they'll submit their public key to an administrator or operator for approval and distribution. Then the operator will then deploy their public key to a host where they need access, sometimes to all hosts or all servers within the infrastructure, maybe using some workflow into the authorized key files of all servers. And then once that's done, the developer now has SSH access from this, from this. From what I've just described, we can already see the first shortcoming of SSH keys, which is that it requires laborious and mundane key management activities. The procedure because let's face it, the process of onboarding SSH keys is time consuming and not scalable for organizations with hundreds to thousands of servers. Like I mentioned, you're going to have to maintain and update keys on the authorized key file of every server, and that can be operationally challenging and intensely time consuming for operators and administrators. So one way organizations try to deal with this sprawl, or key sprawl is as they grow, is to use can authorized keys command on servers. This could be a program or a bash script that outputs authorized public key files. So instead of a file, you can have a dynamic lookup of keys every time someone tries to log in, or they use something like ansible to push keys out to all their servers and keep those files in sync. But then even all of this adds very unnecessary overhead. And the truth is, no matter how thoughtfully executed, the task of gathering and shipping and maintaining SSH keys always remains messy. It's always messy. Always how do you keep track of all credentials that can open a door in your infrastructure? Do you have up to date inventory of all SSH enabled SSH keys in your organization? Do you even know who is using ssh in youll.org or which keys belong to which person? Because SSh keys on their own do not encourage a strong understanding of which device or which user owns specific keys. And there's always ongoing cost associated with monitoring and maintaining homegrown machinery for adding, removing, synchronizing, and auditing static public key files across your fleet. Now the second reasons why ssh keys aren't so great for you is that ssh keys do not expire. They remain valid forever, whether you intend them to or not. And the truth is, as long as the public key is in place of somewhere, the private key would work. So even after developers leave your company, their SSh keys would still remain valid. And they or some other person Ssh keys can be shared, theyre can be stolen. So they or some other person or can get a hold of those private keys and use it to wreck know, just like the story of shapeshift, godaddy and other organizations. And the worrisome thing is that there isn't actually any way to revoke inactive keys. So to revoke or remove a key from youll fleet of infrastructure, you have to update all your host or servers to remove that public key. Now, can you easily update all your servers and then back to the tracking problem of SSH keys? Still, when a developer leaves, how do you know the keys, the exact keys that belonged to them? So then, how do you know which key to remove? So unless your company builds its own custom solution, Ssh keys do not support that level of object trail. The next reason why SSH keys are really bad for you is that it encourages unsafe user behavior and bad security practices. Have you ever tried using into a remote host for the first time and encountered the image or the warning on the screen? Now, the default behavior here for most of us is to click yes and just move on with our lives without giving it a second thought, but that shouldn't be so, because what's happening here is the SSH client literally screaming at youll that the authenticity of the server or host you're trying to connect to can be proven like it's telling you that you can actually be sure that you're connecting to the right server or that the server that you're connecting to belongs to your organization or is an infrastructure within your organization. So what's happening theyre is that your client tries to initiate an SSH connection to a server, but the server tries to authenticate the client by checking. The server first tries to authenticate the client by checking if the client's public keys is present in its authorized key file. Now it does that and it is. Then your client also tries to authenticate the server by checking if it is a trusted server or if the server's public key is in its known hosts file. Now it does that and it doesn't find it because that's not ingrained. It's not a process that usually occurs in traditional SSH keys deployment. At this point, the right thing to do here would be to verify theyre fingerprints by using an administrator or an operator or consulting a database that has fingerprints of all servers across your fruits. But the truth is that how many people do that? Yeah, and so when you type, yes, your connection proceeds without authentication and the server's public key is permanently added to the known host file of your client. And this whole scenario is known as trust on first use antipattern. That's tofu. In this scenario, you're basically making assumptions that you're connecting to the right server. And frankly, that's a scary assumption to make because it's a cold world out there. Yeah, and theyre are top of mind, three reasons why Ssh keys are just bad for larger organizations. And then now you're probably wondering, is there a better way to manage SSH? Is there a better way to manage SSH access that makes is there a better way to manage SSh keys? Can we do something about SSH keys to rectify these situations? And quite frankly, I would say that you're asking the wrong questions. The truth is you can't tame ssh keys. Yeah, they just have a rudimentary fundamental design flaw. And so the answer would be to just use SSH certificates and here's why. So what I have here is what an SSH key certificate looks like when decoded. So an SSh key is simply a public key bound to some meta data and policies defining who the certificate belongs to and what the certificates can be used to assess, among other things. And I will just walk through some important properties or some properties of the SSH certificate on screen just to give you an idea of what it enables. So first we have the type and there are two types of SSH certificates. We have the user and the host certificates. What I have here is a user certificate. So host certificates are used by hosts or servers to authenticate themselves through SSH clients. I will talk on that later when I describe how SSH certificate authentication works. And for user certificates, which is what I have here, theyre are used to authenticate SSH clients to host. Now, user stats are quite similar to host. The major difference here is the principles field, which we'll get to next. You have the public keys, and this is the same public key portion of the key that you would have with a traditional SSh key. The difference with a certificate is that a certificate authority, which we'll get later, has signed all of this metadata that you're seeing and satisfying that the public and private keys pair is trusted within your infrastructure. Next we'll have the signing CA, which is the signature of the CA that signed, which is the signature of the certificate authority that signed this certificate. Then we have the key id and this property is usually outputted in SSH logs so you can put any important identifying information here. Then we have valid which defines the lifetime of an SSH certificate, because SSH certificates do not live forever. Next we have the principles. And principles are an arbitrary string that allow access to a specific host or a specific user on a specific host. And then we have the extensions property which allows for privileged SSH features like agent and port forwarding, or even to force configuration directives. Now let's look at how SSH certificate authentication works. Now to get started using SSH certificates based authentication, the only additional element you need to introduce to your infrastructure is an SSH authority. Now I make the distinction of calling it an SSH certificate authority because you cannot use a certificates authority configured to issue S 509 or TLS certificates to issue SSH certificates. And that's because OpenSSh uses a custom and simpler certificate format. Now as I mentioned in the beginning, SSH certificate authentication works almost the same way as using SSH keys. Now in this case you would configure clients and hosts to trust certificates issued by your CA. And to do this it would involve something like editing the SSHD config file on each host to specify the CA's public key so that that can be used to verify user certificates and then youll can also add. It would also involve you adding your CS public keys to each client's known host file to be used to verify host certificates. I won't go into the technical details of how this is set up because that's not what this talk is all about and different tools have different ways to do it. And some tools make this process automated. Yeah, I'll link to resources at the end. My duty theyre is to convince youll to consider using SSH certificates in your infrastructure, show you why they are great, and so you can consider using them in your infrastructure if you're not already. So as you can see in the image, when an SSH client tries to connect to a remote host, it first obtains a valid certificate from the CA and the host will typically get this host certificate when it is first deployed and then be renewed regularly from the CA, maybe each month on a timer, then also once. Then the user will also ask for a certificate and signed by the CA. Then when the client tries to connect to a host, the host asks for certificates from the client and confirms that it has been signed with the private key of the CA that it has been configured to trust using the CA's known public key. And then the client will also do the same and ask for a certificate from the host and verify that the host certificate has also been signed by the public key of the CA that it has been configured to trust. And then once certificates have been exchanged, the protocol would then proceed as it would with public key authentication. Now this brings me to the first reason why SSH certificates or using SSH certificate authentication is far superior to traditional SSH key authentication. And that's because the way SSH certificate authentication works immediately eliminates the trust on first use anti behavior that we talked about. So youll can have both clients and hosts can be sure that they are connecting to infrastructure that actually belongs to their organization. It creates a walled garden, which is great for organization. Another great thing about using SSH certificates is that they are easier to maintain and manage. That's after the initial setup. Static keys in authorized key files are no longer needed, and so you don't need to have scattered keys across your infrastructure. Instead, peers, clients, host, and clients learn one another's public keys on demand when connections are established. By exchanging certificates. Now, eliminating key approval and distribution processes has immediate operational benefits for organizations because you're no longer wasting up cycles on mundane key management tax, and you eliminate any ongoing cost associated with monitoring and maintaining homegrown machinery for adding, removing, synchronizing, or auditing static public key files across youll hosts. Now, also, because certificates allow for inclusion of metadata, you know exactly who a certificate belongs to. And then you can implement more fine granting access controls and you can have youll visibility. And when you have visibility, visibility makes way for things like governance, compliance and reporting. Now, the third and probably the most important reason to use SSH certificates is that SSH certificates expire and they can be revoked. You can define a validity period using the valid property that I talked about. That I talked about. So this little detail is so crucial because it encourages safe security practices and reduces companies risks, exposure and surface attack. Because unlike traditional public key authentication where key management oversight and that's like forgetting to remove an S employee's public key from a host results in SSH failing open and allowing unauthorized access without end. You can set up SSH certificates to be valid for say 16 hours or longer or even shorter, depending on the exposure risks of the host that they will be used to assess. So even in an incident of a mistake or TEF or misuse or key filtration of any form, compromised SSH credentials or certificates will expire automatically without intervention. So before the attackers would have time to make sense of what they are doing, their assets would aspire. And that's what's so great about SSH certificates. Now, with all of this that I've shared, I hope that you're starting to see how SSH certificates can make your workflow secure, easier to operate, easier to manage, easier to automate. Now the next thing is to get into tooling. How can you start using SSH certificates? Because I do hope that from the little I've shared of how SSH certificates are superior to traditional SSH keys, that you're convinced to give SSH certificates a second thought and a deeper look, and try to discover how you can start adopting them in your infrastructure. Now there are a couple of open source tools that you can use. We have SSH keygen, which can generate root certificates and sign host certificates. There's also Netflix bless which is Netflix SSH certificate authority that runs in AWS Lambda and uses IAM. Then we have n Sheridan cashier, which is intercom's SSH. Can you have Uber palm SSh which lets you use SSh to authorize pseudo use? And then we have our very own smallstep labs step can. Now, I talked about SSH certificates being short lived. And in addition to that, I'm not very familiar with how other tools on dislikes works, but with small step can the private key and certificates always go into the SSH agent and nothing is stored on Dix. And so once the key and certificates is in the SSH agent, they can now be used for connections but theyre cannot be exported easily so it makes it difficult to steal SSH certificates. So in addition to SSH keys being easy to manage, having inclusion inclusivity for metadata which encourages more visibility and better audit trail, and not being short lived and using able to be revoked now, they are also much more resistant to theft. Now, I have to mention that the caveat of open source tools is that you have to do some setup yourself. You might have to probably build the foundation is there, the support is there, but then you now have to build out tooling for dashboard, for visibility, for auditing and so on. And that's why I'm using to mention that small step has a paid SSH product that offers complete lifecycle management, audit login and full visibility into all of your certificates. And you can manage and keep track of everything from one admin panel out of the boss. You just plug in and be on youll way. So I'll link to resources to learn more about it at the end of this talk, just to show you what SSH certificate authentication enables, let me just blow your mind for an instant. I really hope that your mind will be blown. In addition to all these advantages that I've talked about today associated with using SSH certificate authentication, SSH certificates authentication also allows for extensibility. Rather, you can issue SSH certificates via a variety of authentication mechanisms to facilitate better usability, which makes obtaining SSH certificates simpler and makes obtaining SSH certificate a more simpler and elegant experience for your users. So for instance, you can use, for instance, so do you know that you can use SSH with SSH? And that's so awesome because just imagine this, a developer starts their day and they want to ssh into a host. They type SSh username into a client and they are taken to a screen requesting them to sign in via your company's or your organization's identity provider. And once the developer completes their SSO sign in, a better token or can OIDC identity token is returned to the login utility. Now using this token, the utility generates a new key pair and requests a signed certificates from the CA. So then the developer is then issued an SSH certificate with an expiry time, long enough for a workday, say 16 hours or 20 hours, and the certificate is stored in memory and now they can use it to assess SSH hosts as usual. How cool is that? You bring youll IDP and we bring the ssh and mash it together and you have a seamless SSH experience for your developers and users. Now this is what small step SSH does, and by default, small step SSH means a 16 hours certificate to cover a workday, and subsequent SSH sessions won't require reauthenticating until the certificate expires. Now, with a seamless OICD sign in flow like this, it's easy to enforce your existing security policies like multifactor authentication or using Fido U twelve security keys. And the cool thing about this is that removing a user from your identity provider automatically terminates their SSH access in seconds. I mean, if you ask me, that's like the coolest thing. So now when a developer leaves your company and their IDP access is revoked, they lose all SSH access. So you no longer have to worry about having rogue or unidentified SSh keys lying around in your infrastructure. That's no longer a threat. Yeah, no more unused keys using around and needing to somehow get keys off every host, that's no longer a threat or a problem. And still talking about small step, we also have can integration with a tool called indent and it lets users request access to company resources that they need to do their job of Yaslac. And this is just to show you what's possible. There is so much more that is possible with using SSH certificates. So by adding intense on demand access control to small step, now approvers can approve and review requests and initiate certificate generation process right from Slack. And this is helpful in cases where external consultants not in your organization need to get access, or you want users to just be able to request access to SSH into hosts on demand instead of mapping host to users from your identity providers. It's super awesome. And I mean, do I need to say more? Honestly, do I need to say more? I don't know about you, but at this point, if I were listening to myself, I'll probably be sold. Yeah, like I mentioned at small step, we're convinced that SSH certificates are the best way to do SSH, and they are not hard to understand. The initial setup might be a little bit involving, but it's well worth the effort. And I believe that SSH certificates demand more press and broader use. And so at this point, I really hope that you consider using SSH certificates in your infrastructure and in your organization. So you can either use open source step ca SSH enabled certificate authority, you just have to knock out a few things on your own, and we even have a video tutorial that you can follow to do that and set that up in your home lab, or to test it out and then for your organization. Or if you want to get a lot of futures right off the bat, you can check out SSH pro and take it for a spin via a free trial. And as I mentioned, here are some links that you can check out to learn more about how to start using SSH certificates in your infrastructure today. And that brings me to the end of this talk. I hope I have convinced you and not confused you that SSH certificates are the right way to do SSH authentication. Thank you,