Securing Teleportation

Video size:

0:08 Miko Pawlikowski

Hello and welcome to Conf42Cast, another exciting episode today: 'Securing Teleportation'. My name is Miko Pawlikowski and my guest today, Andrew Lytvynov, security engineering team lead at Teleport. Hello, Andrew, how are you doing today?

0:27 Andrew Lytvynov

Good. Thanks for having me, Miko.

0:28 Miko Pawlikowski

We typically start by confusing our listeners a little bit. So your question for today is, what's your most favorite thing about space travel?

0:39 Andrew Lytvynov

Admittedly, I'm not a huge fan of space travel yet, only the sci-fi versions of it. But you know, when you go up to space, you get stuck in this metal and plastic can for weeks and months, possibly, and everything you bring with you, it's all you got. You can't just pop off to a nearby planet and get some snacks. The thing that I like is that you are kind of forced to recycle everything. Anything you bring with you, any hard materials, metal, plastics, what you have would has to be reusable. Anything like food, you have to be able to compost, maybe grow something from it again. You have to filtrate your own urine and drink it again. The constraints of space travel force you to be very, very resourceful, to use the resources you have.

1:23 Miko Pawlikowski

That's a very good answer, especially for someone who prepended that with, 'I'm not so much into space'. You're right that Amazon still doesn't deliver into space, although with Blue Origin making progress, we'll see about that. Hopefully, soon enough. We had Ev coming on our episode that we spoke a little bit about Teleport - how it started and what you guys are trying to achieve with that and the unique proposition. So I wanted to talk a bit more about security angle of that. You do have security in your job title. And from what I saw on LinkedIn, you've been there for a little while. Let's start with the big one. How do you secure something like Teleport, that's kind of like all about security.

2:07 Andrew Lytvynov

So I'm gonna have to prepend everything, as a security engineer, that there's no system that is secure. You can't build something that's 100% bulletproof. But with Teleport, we try our best to get there. And we do that by following some guiding principles, some very fundamental theoretical security principles that have been proven to be pretty reliable over time. One of the major ones is just issuing short-lived credentials in general. In our case that's certificates, but can be anything and being able to rotate everything. So this is kind of a bit of a defeatist view on security. But if something or someone is compromised, at least this time constraint, and you can rotate everything, and you can recover from any issue. Of course, it's not something we just rely on, we don't say like, 'Okay, well, everything's going to be broken and just keep rotating'. So we also do a fair amount of security design review. So internally, and also we contract on external audit firm to do penetration tests, to do design reviews for us. So basically, there's a lot of eyeballs on every single piece of functionality in Teleport to make sure that it's as secure as we can get it reasonably. And the other thing that Teleport specifically tries to take care of, compared to a lot of the alternative solutions, is making setup of everything very easy. If users are not able to configure your authorization policies correctly, if they're not able to set up authentication and 2FA, none of your security really will work out, because eventually someone's gonna get confused. And just misconfigured something and open all the backdoors. So in Teleport, we really emphasize user experience a lot and try to make all these concepts, all these very gnarly security problems as easy and transparent to users as possible. There's very little room for user error.

4:12 Miko Pawlikowski

Okay, so short-lived certificates is of use, additional eyeballs. I'm surprised he didn't mention the open source aspect, because I personally find it interesting. Do the extra eyeballs of people browsing GitHub help with that too, or would you say it's not the primary driver?

4:33 Andrew Lytvynov

To be honest, I know this is a common belief about open source that there's all these people scanning all the GitHub projects all the time and finding all the security bugs. It's just not the case in our experience. There's a few users occasionally, they will find some issue or bug with Teleport, report it and then some of them will actually dig into the codebase and say like, 'Oh, I found what the culprit is, and here's what we should be doing'. But for any non-trivial projects like teleports, it's just too much code for kind of stranger on the internet to understand and dive into and find all the issues. It is helpful to be open source. And it definitely forces us to not rely on any sort of security by obscurity. But it is not as beneficial, as you might expect.

5:21 Miko Pawlikowski

And there is also the dark aspects to it, right? In theory, it will be possible for some not particularly nice party to go and try and plan something in your code base, right? So, I guess, with a project of so much of security as a top priority, how do you deal with that aspect?

5:43 Andrew Lytvynov

Yeah, that's a very good question. And it is a very interesting attack vector, especially the recent past. In terms of malicious contributions, that part is not too hard to filter out. We tend to be very picky about what contributions we accept, and we very, very restrict that code review. So in terms of slipping in something into our code base, it's very unlikely just because we enforce code review everywhere, we do a lot of automated testing. And obviously, we just have senior engineers review any external contribution. But the other possible vector, which we actually had us care about recently, is through your CI pipeline. So when someone submits some innocent looking code, but then for example, modifies their CI configure, but leaves in the same repo or modifies your test code to start, like trying to mess around, break up the sandbox in your tests running pipeline. So that part is harder. And we did have to recently basically rebuild the entire CI pipeline from scratch, because we internally discovered there was a sneaky way to get in, like get in and escalate privileges from that angle. So that part of open source is scary and difficult. And I mean, if you look at product announcements from most of the CI platforms, in the last couple of months, a couple of years old, they were like, 'Whoa, everyone's mining crypto in our tests runners all the time, let's add all these extra like rules and restrictions'. So that is not as bright and shiny and nice as we would hope. And there's a lot of bad actors out there who are trying to, like make money any way possible.

7:19 Miko Pawlikowski

And there's also another vector that comes to mind when you talk about it is the dependencies of your dependencies. It might be easier to slip something into a dependency of a dependency of yours. So how do you deal with that? I mean, you can't possibly review all the PRs of all of your dependencies at all times, right?

7:42 Andrew Lytvynov

It is one of the areas that I've been personally very interested, even before Teleport when I was in GKE. It is a scary, scary aspect of open source. And people are just starting to discover, I don't know, like if I'm sounding very anti-open source software, I'm not. But it is something that both security researchers and attackers and like the rest of the industry have started to realize, it's a big, big gaping hole in the security posture of pretty much every project. Even if you are not building anything open source yourself you, as you pointed out, you will most likely depend on something out there and pull in some code from strangers from the internet. So in terms of Teleport specifically, it's not super scary, because we use GO we use GO modules, we vender all of our dependencies. So like, no one can just update something on there and and it will automatically be injected into our code. We do occasionally, like every release, we will try and update most of our dependencies to the newer versions and pick up all the bug fixes and whatnot. And that partly try to skim through the code, is not 100% reliable. It's theoretically possible someone would slip something by and it's very obfuscated, but we do our best. The other thing we do is being very picky about which dependencies we include and take on. So we will actually evaluate or look at who the owner of the repository or dependency is, how they manage it, is it being updated has been any changes in the last couple of months at all, or is it totally abandoned. So we will try to at least not rely on something that is completely sketchy running, you know, from some guy's basement in Turkey that he doesn't even care about anymore. In terms of transitive dependencies that's more difficult. Again, luckily, thanks to GO, we don't have that problem as much because all the transit dependencies are also vendered and visible and included. You know, in some other languages, it's not as observable, but it definitely could be improved. But Teleport is in a pretty good state, in my opinion, in terms of dependency injections. Now the industry as a whole is starting to kind of tackle that problem as well. There is one group called 'The Open Source Security Foundation' run by the Linux Foundation. And basically, their whole goal is to secure supply chain risks, and to secure all the critical projects that the modern world depends on. Things like you know, Linux, Kernel, all of the very, very low level libraries and systems that run everywhere. That thing is very much a work-in-progress. And I'm kind of keeping an eye on it. But it does look like a lot of the big players, big companies realize the problem and invest a lot of money and dedicate staff to this, trying fix it. But right now, that's probably the biggest risk to any project in the world.

10:42 Miko Pawlikowski

So, you mentioned GO and I wanted to go there for a minute. I think I saw that on your LinkedIn there, you're proudly a Golang person since 2012. I'm curious what made you you know, pick that as your go-to language? Sorry, I'm gonna stop doing that now. Also, whether 10 years later, you can share some of the not so great and some of the great experiences that you've had with it. And basically, a decade later, are you still liking it?

11:16 Andrew Lytvynov

It is true that I've been using GO since 2012. As part of my day job, it's probably more like 2015 onwards. But still long enough to have all the good and bad experiences. Initially, when I started using it, I was still finishing my degree at the University and also working kind of a full time job writing Android apps in Java, at the time. I was like a very junior developer and it's all Windows, IDs and Java and like very bloated, heavy-built systems and tons of dependencies. Everything's very, very confusing and slow and bloated. And also at the time I started discovering some of the things that will ultimately shape my career. So I started using Linux part time, like trying to use it for development. I started to browse Hacker News, discovered that for some reason, and some of the other more typical modern developer tools. And I was always a big fanboy of Google until recently. Like everything they do must be golden and perfect and can have no faults. Yeah, at the time, I was browsing Hacker News and I saw the announcement for Google 1.0 came out. And opened it up, read through some docs, started the the tour of GO and went through that pretty quickly. I was like, 'Oh, this is very refreshing. It's so simple. It's so little boilerplate, compared to Java. It's just so much more pleasant to write'. And also without losing any of the kind of safeguards not losing any of the type system, and kind of the things that come with a mature compiled language. So after I figured out that I liked it, I started using it for some University projects and assignments. My professors were pretty receptive to that. After building some experience that way and doing some side projects, I found my first job, which was in the startup in the states that let me use GO for some reason. They were not very picky about the technologies. But it was a very great opportunity. And then I pretty much haven't stopped using it from my day job ever since then. Obviously, as you do something for many years, all of the words and bad parts come to service, you lose your rose tinted glasses. So GO definitely has some problems, I would say or limitations for areas that it's not as usable. Namely, doing anything that involves just quick prototyping, just getting something off the ground quickly, just playing with some idea or anything that involves scripting, it's going to be much more verbose and heavy than just a Shell Script or Python or something like that.

12:41 Miko Pawlikowski

Boilerplate.

12:46 Andrew Lytvynov

And also anything that's kind of related to scientific computing or data crunching, stuff where you deal with a lot of very sort of theoretical code and algorithms and data structure. Mostly that's because of a lack of generics, which I know is going to be fixed soon. But right now, it's still difficult to write reusable performance algorithms that are very abstract. But the same time GO has lots of really good niches that it's in and is basically the go-to language now. So anything that involves network services, like RPC servers, all of the clouds, orchestration infrastructures basically on GO these days and also CLIS because of the deployment model, just kind of toss a binary over and trans everywhere. So those are the areas where GO shines. And really like the limitations I mentioned, just make GO not very suitable for MVPs and quick and prototyping, but any production code you want to write, anything that runs on a server side, it's perfect for it.

15:05 Miko Pawlikowski

As long as you're happy running a garbage collected language, that is.

15:09 Andrew Lytvynov

Yeah. I mean, I wrote some C in the University, but I never really found garbage collector to be such a huge issue that people make it out to be. Especially, you know, as you look at the progression of GO's garbage collection over the years, right now, it's totally a non-issue unless you need something that's real time, where you cannot tolerate a one millisecond pause. There's very few problems that that matters.

15:33 Miko Pawlikowski

And I also hated the GO path before the modules came along. That was a mess. Personal opinion.

15:40 Andrew Lytvynov

Yeah. Now, I would never go back to GOPATH. But I remember back then, when I was kind of hanging out with a bunch of GO evangelists, some GO IRC channel on Freenode, and just kind of helping out newcomers or just solving issues, it was surprisingly easy to deal with in the early days, when you didn't have so many dependencies and so many projects using it. It was not that bad. And it was very easy to explain to anyone. So I don't hate it. But I do appreciate the GO modules and having all your projects nice and compartmentalised.

16:15 Miko Pawlikowski

Okay, I like the story. So basically you were GO-indoctrinated before even joined Google.

16:21 Andrew Lytvynov

Yeah. Yeah. Like when I joined Google, people were surprised that anyone outside of Google use GO like, 'Oh, there's actual GO code outside, how strange'.

16:30 Miko Pawlikowski

Yeah, I want to also mention the GKE. Because for me, the primary driver for GO was justified at Kubernetes, and the entire ecosystem, all of a sudden, everything was in GO. And so if you wanted to read any of that code, you kind of need to speak the native Kubernetes language that is GO. So I was wondering, do you know why they started with go for Teleport? Is it just because it was the coolest one? Or was there any particular technical requirement that made it better than any other language?

17:01 Andrew Lytvynov

I was not at Teleport from the very start, so I don't know the context of a lot of the technical decisions from back then. But I think all of the founding team were ex-Mailgun employees and at Mailgun at the time, they adapted GO kind of early on for some of their internal projects. And it seems that they liked it. So I can't tell you the exact specific reasons why GO was chosen against anything else. But it seems to have worked out very well. Especially because Teleport is predominantly a self-hosted product. It's not a SAAS product. Well it is now, but most of the users started running it on prem. So deploying something where they can go is so vastly easier, both on developer machines and on the server side. That makes total sense.

17:51 Miko Pawlikowski

Yeah, that's the real advantage. Okay. So if you were to nominate a single technology, or language or something to particularly follow in 2021 or beyond, 2022, do you have any particular favorites that you think is going to blow up in popularity that will be super useful. The next big thing, what's your pick?

18:15 Andrew Lytvynov

One of the obvious ones is Rust, has been already blowing up in popularity, already cashing out, even surpassing GO in usage. But I don't know, I don't believe it will be as revolutionary as everyone would expect. It's going to be useful, it's going to make for more reliable software, if it's not going to be revolutionary. The thing I'm most excited about is all of the work that's going into supply chain security, which I already talked about, but this is the most interesting topic for me right now. So there's both a lot of security research and just malicious actors trying to abuse it and find all these holes in the supply chain and the dependencies, which have very widespread implications. And there is also a lot of people working on solutions to that. So there's now a bunch of projects coming out of Google around this, things like six store for basically signing and attesting to your dependencies and building a little bit more trust into what you pull in. But also, it's going to force all of the developers to be a lot more disciplined and mindful of what you pull in as your dependencies. Right now, the kind of best practice, if you will, is you just find something that has a lot of good upstarts, you pull it in and you don't even look at what it does, what it depends on. And you do that 100 times over and now you're with 1000 deep dependency tree, that's total unmanageable mass. So with this focus on supply chain security, and you know, the the US government has been kind of looking at it as well and trying to push people, at least the ones who contract with the government, to at least pay attention to their supply chains. I think is going to make for a much less bloated and buggy and risky software. Because people both try to filter out what they pull in and only depend on things that are reliable, maintained, and high-quality. And also, as developers who work on those dependencies are kind of incentivized to pay closer attention to how they build stuff, invest in their own securities. So things like 2FA in your GitHub account and all those things. But more importantly, there will just be more incentives for people to maintain stuff, pardon it and not just toss vacant project out there in the internet and forget about it forever, because no one's going to use it anymore.

20:36 Miko Pawlikowski

Very fair points. Alright, I'm gonna ask you one last question. And that's the one where I tend to try to collect the golden nuggets of wisdom from people. If you were to pick a single thing that provided the highest return on investment for your career, and can be anything from a habit to skill, book, anything really. What would you pick?

21:03 Andrew Lytvynov

One thing that I learned in the last maybe five years and has been the most useful, it's a simple thing. But a thing that is never explained or adviced to junior engineers, sort of like most engineers, and is just asking questions. So whenever you're in a room discussing any technical topic, if there's any point of confusion, anything that you think you kind of intuitively understand, but not really in detail, or you don't understand the motivation for why something needs to be built or fixed, it is very, very valuable to ask, regardless of the rank of people in the room, or how many people are there. You can't appear to be stupid if you ask questions.The odds are, if you have multiple people participating in discussion, some of the other ones that are also being confused by the same thing, they're just too afraid to ask. When you ask questions, it's going to give you a lot more understanding and context on everything that you work on. It is also going to help you build relationships with your peers, because the conversations can be much less dry and you're going to get to understand their reasoning better. They're going to understand your reasoning and way of thinking better. And it's just generally going to accelerate how quickly you gain experience on your job. So that would be my biggest advice. Never be afraid of asking questions, even if they seem dumb.

22:21 Miko Pawlikowski

Absolutely. No such thing as a stupid question.

22:24 Andrew Lytvynov

Yep.

22:25 Miko Pawlikowski

Love it. Thank you so much for your time. That was really insightful. I hope that you enjoyed it as much as I did. If you want to follow Andrew, what's the best place? LinkedIn, Twitter?

22:38 Andrew Lytvynov

Probably Twitter, @Andrew Lytvynov.

22:41 Miko Pawlikowski

There you go. And to go check out Teleport?

22:44 Andrew Lytvynov

It's goteleport.com.

22:46 Miko Pawlikowski

Alright. Thank you so much and see you next time.

22:49 Andrew Lytvynov

Thank you, Miko.