Hi everyone, my name is Tal. Thank you for joining serverless security, top ten here at 42 Cloud native. Let's start. So before we start, who am I and why am I here? So my name is Tal, as I said. Now I'm the head of security research of cloud Native technologies at contrast security. I got there through an acquisition of Cloud Essence, a company I co founded that created a serverless testing, security testing solution. Prior to that, I was head of security research at Protigo, another service protection solution who got acquired by Checkpoint at 2019. So I've been doing serverless for the last couple of years and I'm here to talk about it. So I know we've all talked about cloud native in this conference, so I'm not going to add too much of it. But you can see that Forrester predicted that 25 of developers are using serverless regularly already. So I'm assuming one of four of you now, if you join the talk, probably more are using serverless and this is why we are here. Okay, so this is how a typical, actually not so big serverless architecture look like. This is one of the accounts that we monitor at contrast, and you can see a lot of resources and cloud services connected to each other in a very complex way, we can say, but this is not so complex. However, there are many security challenges when dealing with such environment which you don't have a big flow, a server or monolith with one big logic flow, but instead you have a lot of resources and services that are disconnected to each other by nature and then configured to create some kind of a logic with each other. What is a serverless environment is can event driven architecture. Actually, when you have in AWS, for example, you have lambda functions which basically host your code and run your code, execute your code based on cloud or events in your cloud. Events can be API gateway, so rest APIs, HTTP request could be table changes, Iot rules, logs at three buckets of files or anything that you can think about. And those functions really run your code to interact with other services, typically in the cloud. But because this is your code, this is also where you have create your mistakes, especially security mistakes that can cause some cloud damage. And we'll talk about it. So we said lambda lambdas are really the compute, the service that runs your code. Those are read only environments that have a slash tamp folder if you want to run or to write some files. They are not wired to the Internet in the sense that you cannot ssh to them. They of course have connection. You can create HTTP requests or any type of request, inside or outside, but you cannot connect to the runtime. The data is temporary. Once the runtime or the code finished to execute, it evades, so the data doesn't remain there. It's ephemeral. The code itself resides in the environment. So when the code needs to run, basically AWS in this case, spins up the environment with the code inside and execute it. Also, the permissions for the function, which are the keys translated to the keys in the runtime, are located in the environment as environment variables, so the function can interact with other services based on its permissions. All right, so we said we're going to talk about this security top ten, so whatnot. If the main project OS serverless top ten project that right now interprets the original top ten security top ten, running an open call for a few years, there is little data, so we're waiting for more data to come up. To create a serverless tailor top ten, you're more than welcome to contribute to it, right? So let's start with number one event injection. Like any other injection attacks that you're familiar with from web technologies, pretty much the same. The difference here mostly is the entry point. So it can come not just from an HTTP request, but by many uncontrolled entry points. And we'll talk about that. We mentioned a few of them, like s three so files, emails, IoT, and we'll talk about it in a second. Traditional attacks still exist like injection command injection, SQL injection or NoSQL injection code injection, depending on the runtime of the function. And there are new types of injection like email injection, which we'll see later on MqtT pub sub. So things that you weren't really familiar with, and even voice command injection, the impact depends on the function's permissions. So if I have an injection attack, the impact that the attacker can cause it depends on what the function can do in the cloud. So few entry points, so rest APIs, of course, maybe the most common ones. But we can also have three party applications connected to our application to our functions, cloud storage, authentication services, logs and events, IoT, voice command via alexa, search for it on Google, you'll find something, email, SNS, code pipelines and many other things. Anything basically is a lambda or the function can be connected to. So what are the best practices here? Well, like traditional application, never trust, pass or make any assumption regarding input. The problem here is that we don't have security controls in the perimeter because actually with serverless we kind of lost the perimeter. We don't have the network access to the entry point to the function. We don't control it. Basically when we run function we just can the code. So the code has to secure itself. So we need to use positive like whitelist input validations inside the code. API gateway can also allow configuring some JSON models to limit the type of request. But you have to make sure your code can secure itself. You have to consider all event types and entry points. If before voice command injection owasp not something you should think of now you should. And of course just to limit the impact, you should run functions with list privilege managing just the required permission for the function. So of course there are some commercial tools that helps you protect your functions in runtime or detect security. Best practices shift left basically second part is broken authentication we've seen before the image with all the resources connected to each other. So the functions are really stateless, they don't really have a session to them. So there are more tricky ways to control the authentication. But basically think about zero trust functions cannot really trust anything, they just execute the code. You can store a session out of bound like a database, but I wouldn't say that's a common way nor that's a best practice. You could have multiple entry points, services, events and triggers that come into your code with no continuous flow to control the authentication. For example, here you can see a lambda function which is maybe secured by itself so you cannot reach it directly. But there is can open sns here, an open ses here that triggers the function. And if we can get directly to the ses, we can actually run the code we as an attacks. So you have to make sure the function can secure itself without even knowing who is going to send the email. In this case best practices. So if you're using AWS, use cognito whenever possible you can use of course other authentication services like Okta. Depending on the service on this cloud that you're using, use access tokens that can include custom data and can be signed jot tokens. For example, if you need external user perform input validation. Run with list privilege is always a best practice and just in case if you need out of bound security state. But I'd say the best practice here is to be able to run with zero trust. So when the cloud runs, it already knows in a way that the user was authenticated. So you must make sure of that. Number three, sensitive data exposure. Well that's really same as any other cloud based data. Our data is stored in the cloud, so we need to protect it in a different way than on our say. It doesn't mean it's less secure. In many cases, I'd rather my data to be stored on databases and be managed by cloud providers like Google Azure or AWS rather than a small company. In terms of customer data, when we execute code, the data is stored under temp. We also have other sensitive data like the environment variables that can contain tokens, access keys and other secrets. Other types of sensitive data, which is very common is data in buckets. We've seen many times exposed f three buckets which breach that expose data that was unsecured or put into a cloud storage. But the configuration of the cloud storage left it open for hackers to access. Source code, as we mentioned, is also in the environment. So if someone can run access or to gain access to your lambda function, they can also have access to the source code. The best practice. So if you're using a lambda and you store data into slash temp, delete it after the use so another execution won't have access to it. Use kms the key again, if we're using AWS, kms and equivalent to encrypt environment variables to encrypt data, and you can use also secret storage if you need to transfer data secrets between lambdas and services. If you're using other services, make sure you're securing them. For example, buckets don't make them public. That could be a huge risk for your company. Again, if the lambda handles sensitive data, make sure you run with less privilege. We'll hear it again and again because it limits any type of risk, and we'll talk about it independently later. And you can use other services. Again, if we're running AWS, you can use Macy, for example, to identify sensitive data in your buckets. All right, so we talked about list privileged a lot. So overprivileged function is number four. In this case, I'd say that more than 90% is an understatement. I'd say more than 95% or 99% are misconfigured. The difference between small mistakes and big mistakes. That's the difference. Functions tend to use many permissions that are not required. The reason for that is that because it's hard to configure a very tight security for each function for many reasons. Maybe we don't know exactly what is their right permission, or maybe lack of time, or could be other stuff. In many cases, security teams inside organization even give the developers predefined roles, which has many functionalities and many permissions that they don't need, but they tell them to use them. So the developers are really limited to use those. The impact of other vulnerabilities and other security risks that we discussed again depends on the permissions of the function. Because if the function only has permission to execute the code, and I don't know, write to the log, to the STD, out to the log system service, then the impact here, even for a code injection, is quite limited. On the other hand, if the function has permissions in many cases like wildcards, to services, to add three buckets, and we will see that in a demo later, you can see how big of a mess it can do. Okay, in many cases it's going to be limited, maybe stealing some data from database or from your cloud storage. But in extreme cases it can really lead to a disaster into leading to your complete cloud account takeover. Here is an example. So the last part on the left is an s three permission that has really a wild card both on the action and the resource. In the middle we've added some security, which is adding the specific bucket, but on the right side is the list privilege. We can see the getobject and the bucket name. So this really limits just what we need into the function. And again, best practice, review each resource is function and apply list privilege. There is almost impossible to do manually unless you have a handful of functions. So you have some way to automate that service, that thing, and make sure each function that goes into your production has its list privilege, right? Number five, vulnerable dependencies. Of course, this is very common. Using dependencies can be insecure if you're importing dependencies with security vulnerabilities, because functions tend to be very small in types of the custom code that you write, they tend to bring a lot of dependencies, right? So a lot of imports. In that case, it may lead to security breaches in your code. Again, depends if you're actually using the vulnerable part of the imported dependencies. I probably don't need to explain that to you, but when you import a specific library like request, in this case it brings with it the whole family and everything, all the dependencies of the dependencies and the dependencies of the dependencies and so on. So best practices, scan your dependencies before deploying into production. Use open source or third party solutions that can make sure if you can use secure versions whenever possible. Replace library or apply your custom patch if there is no security patch outside. Number six, insufficient logging and monitoring. So why is it different here? Well, because we're using serverless and cloud technologies. There is a logging and monitoring service by default in AWS, but you need to know how to use it. So it's a little bit more complex because you need to know how to extract data from it, how to access it, how to automate the process, because of course you cannot manually go and check. So basically if you have a production environment with millions of invocation, it's really hard to understand or learn anything from that. All right, because of time limitations, we're going to see a demo now and then we'll cover really quickly the other security issues. So as you can see here, we have a user that sends an email. The email then triggers a lambda function number four. And the function does two things. Store the file that was uploaded to an athlete bucket and then reply directly to the user. So the user doesn't have direct access to the function. Right, but the email service does. In this case we'll see a demo, how the hacker, in this case the user send the malicious file into the lambda function, which has of course a vulnerability. The vulnerability would then impact or cause the lambda to run malicious code and change the security configuration of the bucket, as well as submit emails to other destinations. Okay, let's see. So in this case this is a call for papers system and the user is going to send an email to the CFP system, sending an innocent PDF of course, just to check the system and see how it works. And as you can see, after a few seconds we get a reply. Thank you for your submission, Defcon. Of course. So in this case what I want to do is send a malicious file just to see what happens. Now, just a regular malicious file. The PDF that we've now submitted has some malware in it and we've got two emails. The second email which says we've identified malicious content in your file and there was a link to Verus total. Okay, so now what we're going to do is well, let's try to exploit the function that does the security verification. Okay, so I pre wrote a security code that will be downloaded by the function and will send emails to different destinations that I want. In this case I'll reply to myself, but I can send to whoever I want. But the best part is I'm going to change the security configuration of the cloud storage, the s three bucket that holds all the submissions, all the CPF to a public bucket, and then I'll be able to access all the submissions of all the users. Let's see how that plays. Okay, I'm uploading the file to my own bucket, just so I'll be able to access it through the web. The lambda will be able to access it through the web. Then I'm going to send this file with a malicious file name that we're going to exploit. A command injection in the lambda tells it to download my file from that bucket and execute it, uncompress and execute it. And just for fun, I've added some requests. This is the s three bucket that holds all the submissions. As you can see, I don't have access when I'm trying to. Okay, let's send the malicious file then. Here on the right side you can see that I've got some notifications. I just wrote some to myself, some notifications, so I'll see that things are happening in the background. And as you can see now, I got two emails back. One is the regular one, but the second one is part of the exploit saying, hey you, congratulations, you were accepted to Defcon. And here is your honorarum of $300. So click here to gain access to that money. But this is of course a phishing link which I can send to other participants. And this is the first part. But the nice part, the second part is here where the bucket was not accessible before, is now fully accessible. And I can see all the submissions. I can also download them to my own computer. Okay, so we're done with the demo, let's continue. So we've covered six of the top ten. We have four more. One is open resources. So we've talked about broken authentication, but we can actually have just open resources like queues, like SNS services, like s, three buckets, like API gateways. Those can be just misconfigured to be open and can lead to, in this case, security issues. The next part is denial of service versus denial of wallet. So lambdas have limitation of 1000 concurrent lambdas at the same time for all the lambdas in the account. If I'm limiting that, I can have denial of service even if not because someone can actually send more than 1000 requests in parallel. But in any case, I'm going to have this debate between denial of service and denial of wallet because execution of lambda functions cost money per execution. So if I'm not limiting the user, and the function can be accessed from unauthenticated user or for a registered user, then they can cost money. If they leave multiple requests overnight or over one week, I'm going to pay for it. So yeah, the payment is by the millions, but if it's a targeted attack, it can cost me a couple of $100 insecure shared space. So we've talked about the slash temp. So between executions, if I'm running in parallel, then they of course don't share the same runtime. So they don't share the same data. But if I'm running a request after a request for performance reasons, performance reasons, the runtime is recycled by the cloud provider, leaving the data inside. So if I have access to slash temp from the outside, if the developer didn't delete the previous execution, I can access it and I can steal data of other requests coming from maybe other users. And the last one of course, is insecure secret management. Again, this is some kind of a complex because lambdas are independent, stateless and ephemeral. And it's kind of hard to gain access to store data. Developers tend to store data hard coded. This is very unrecommended. You can store data in your environment variables if you encrypt them. If not, again, it's a risk. There are other services that allow to set place like a vault that you can extract sensitive data from like secret managers or really vaults if you're using terraform or something like this. Hashicorp okay, so we've covered the top ten. We have, at contrast, our own method to do that. We're doing it not by protecting the function, but by testing the function for security vulnerabilities to remove the risk, including list privilege. So we connect to an account, we scan everything, we monitor everything and everything that we find on your lambda functions and the connected resources like APIs, API gateways, and f three. We can detect misconfiguration and security vulnerabilities in your code. In your custom code we can detect functions with over permissions, and in that case we can secure your serverless environment. This is an example of a list privilege permission, so we can detect that the function needs only one specific permission. In this case was DynamoDB scan. Okay, if you want to learn more about security in serverless environment, you can use DVSH, an open source project owned by OASP that lets you both secure and play the offensive part. In a service environment with just one click you can deploy it, but make sure you don't do that in production because unless the account is native to that environment, you can have some problems. Okay, thank you very much. Tal, shoot me an email if you want to discuss more about serverless and serverless security.