Conf42 JavaScript 2024 - Online

- premiere 5PM GMT

This DoS goes loop-di-loop - preventing DoS attacks in your Node.js application

Video size:

Abstract

Node.js’ single-threaded nature makes it very susceptible to DOS attacks. In this talk I’ll cover some common mistakes that could make a Node.js application may vulnerable to DoS attacks and some common best-practices to defend against such attacks.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Good morning, everyone. Welcome to CONF42. My name is Alon Mironik and I work for Blackduck, where I manage the R& D efforts for the Seeker agents. Seeker, in case you haven't heard about it or rather haven't heard about it yet, is probably the best IAST tool out there today. But as fascinating as IAST is, it's just not my topic today. So this is the last time I'm going to mention Seeker. Instead, Today, I want to talk about DOS attacks, what they are, and how we can prevent them in our Node. js applications. Now, if you're my age or older, when you hear DOS, you're probably thinking of a really outdated operating system. One would argue that running your Node. js application on this operating system is a vulnerability as of, as in itself, but this isn't the type of DOS I'm talking about today. Today I want to talk about denial of service. For those of you unfamiliar with this concept, let's go straight to the source and see what OWASP has to say about this. So the denial of service or DOS attack is focused on making a resource such as a site, an application, or server unavailable for the purpose it was designed. And of course, a lot of you, when thinking about DOS, instinctively think about DDoS. distributed denial of service, which is an attack where you hit an application or server or whatever with multiple requests from multiple sources and essentially crash it or bring it to a grinding halt just by the sheer volume of requests. This is a very real problem. And a very interesting topic to discuss, but since we are in the JavaScript conference today, I will not be talking about DDoS at all. Today, I will be focusing on the application. What can we do from the applicative side in our Node. js applications? to prevent for argument's sake a single request from creating a denial of service or DOS effect. Now, this won't fully solve all of our problems. Our applications may still be vulnerable to distributed denial of service attacks, but by making our application secure from DOS attacks, first of all, we are reducing the attack surface for distributed denial of service attacks. And secondly, we try to follow the principle of solving the right problem in the right place. On the application side that we will discuss today, we will see how to make our application resilient against a single request that causes a malicious denial of service. Solving distributed denial of service attacks is usually done not in the application level, but in the deployment level, with rate limits and size limits and load balancers and things like that, which again is just not the topic today. And we will not focus on that too much. let's reel it back in and let's focus on the application. Specifically, our Node. js application. let's start with a quick reminder on how Node. js even works. And this is a gross oversimplification. There's actually really good documentation in the Node. js project about this. So if you want all the gory details, please don't take my word for it, but read up yourself, but very briefly and very oversimplified node is designed to scale really well. The basic design for node is a single main thread called the event loop, which should handle all of your user requests. And the design tailors to applications where you have really small chunks of work on the event loop, and anything that takes a longer time, usually IO, but not necessarily, is delegated to a worker thread, which is then executed by a limited number of threads out of a given worker pool. Now, this design helps node, excuse me, scale in two ways. First of all, by having a limited number of threads, We don't waste system resources on opening and closing and managing threads like older designs and older web servers might do. Second, and what Node got right here in my mind, is the responsiveness. When a request comes in, you want to handle it really quickly, be able to take it in, understand what you need to do, and then delegate whatever long work you have, and maybe respond a bit later, asynchronously. But the idea is that every chunk of work, every tick on the event loop needs to be really small. And that way you can always get a new one whenever a user request comes in, you are not blocking anything. This is something crucial to remember. This design is the core of Node's performance. That is the core of why Node can scale this well. And, unfortunately, it's also the core reason of why Node can, Node. js applications at least, can be really vulnerable to DOS attacks if you don't write them the right way. Just to hammer this point back home, a lot of people when they think about DOS, they think about performance. Thanks. which is a good correlation to make. But preventing DOS attacks is not about speed, or at least it's not just about speed. Of course, however faster your application is, it's more difficult to create a DOS attack on it. But the core point here isn't slowing things down, It is blocking other requests, legitimate requests, from being served. So if a malicious request takes two hours to fulfill, that might be okay. The core issue here is not that it took two hours, but whether during these two hours, we are able to serve legitimate user requests. how can we block a Node. js application? How can we tie up the event loop? it sounds really fancy and really dangerous and really far fetched. It really isn't. We will see a few examples of really common practices, things we are used to find in virtually any application that could tie us up if implemented incorrectly. Probably the most obvious example is JSON parsing. We get the JSON, we get, sorry, we get a request with a body. This body is in JSON format. the HTTP client will save that, of course, as a string. And then we want to parse it. In this example, we have the express middleware doing it for us, but that's beside the point. The body is parsed into a JSON, and here we have a simple application that just prints out the number of keys in this JSON. Now, this is a sneaky example. It isn't obvious here. Our callback function in the post handler is running very fast. It doesn't do anything. The sneaky part here is the JSO passing. That happens in the express model way, and the obvious way to overload this middleware is to just send a really large JSON. Now, I was trying to figure out how I can illustrate this and show you how bad this really is. And as a DOS geek, I was struggling to find a way. But then I remembered I'm not really a DOS geek. I'm a DOS nerd. The difference is a geek will tell you about something they're passionate about. A nerd will tell you about something they're passionate about. With a graph. So indeed, I graphed this out. I graphed out the time it takes to serve this request as a function of the length of the body of the JSON in the body. in kilobytes for argument's sake, and we can see it is more or less linear. Now, I know I said it isn't about speed, but again, the key point here is that this JSON passing, which under the hood, under all of Express's bells and whistles, just calls Node. js's JSON. pass is executed on the event loop. however long this takes, It will block any other requests from being served during this time. as an attacker, if I can have my application handle an arbitrary long JSON, I can tie it up for a linear amount of time. depending on the size of the JSON I send over. As defenders, as application authors, what can we do about this? quite a bit. First of all, obviously, if we don't allow tainted input or user controlled input to be parsed as JSON, no problem. This is great. It's not really realistic. JSON parsing is ubiquitous. JavaScript is in general and Node. js in particular, a lot of applications or services pass information between them in a JSON format. So just saying never pass any string you get from an outside untrusted source as a JSON is not realistic. More realistically, you can just limit the size. A length check is a very cheap operation, and if you can just limit the size and double check the length before you try to pass this long string as a JSON, you have solved a lot of the problem. The key here is to know your applications. Get to know how your applications behave. Get to know what a reasonable, valid user input is. Maybe Add some buffers for extreme cases, but limit the size. Don't allow the application or the model way in this case to just pass arbitrary long strings. This is a real example I saw in a customer a couple of years ago. They had a very straightforward login screen. You type in your username, your password. The front end takes these values, puts them together in the jsun, sends over to the backend. The backend either authenticates or doesn't, and sets a cookie. They had eight character usernames and I think passwords were limited to 12 characters. Let's say the JSON format has some overhead. For argument's sake, 40 characters together, 50 to handle extreme cases, and I don't know what. Why in the world would you allow your backend for a simple username password login screen to parse a 2MB long JSON? Absolutely no reason. The only reason to do that is just carelessness. It's forgetting to set a size limit. set your size limits. Where you can't set your size limits, where you do expect really long, big JSONs, because this is the protocol, this is how your application shares information. Maybe don't do it on the event loop. Don't use express. json. Definitely don't use json. parse yourself. But use a different library that can process large JSONs. In the background as an asynchronous task libraries like JSON stream or BFJ that stands for Big Friendly JSON and no, you will not convince me the F stands for anything but friendly. Now, just to hammer this home again, this is not necessarily fast. Using BFJ for argument's sake is explicitly slower than using JSON. parse. They even say so themselves in the documentation. But, speed isn't really the point here. We don't care if this parsing takes a bit longer, as long as it's done in a worker thread and does not prevent us from accepting or serving the next request that comes in. But okay, enough about JSON. This was boring. An attacker can only get a linear effect to the size of their input. It's a threat, it's not a terrible threat, and it's still pretty hard to do damage like this. Let's look at something a bit different. Let's say we aren't just overloading our parser, we're bombing it. And the classic example is XML. So again, I have a very simple application. It takes a request, a POST request, retrieves the body, parses it as XML, and returns the number of child nodes in the payload. Thank you. Not super interesting, but I threatened that we are going to bomb our parser, which sounds quite serious, actually too serious. It's frightening. So let's relax. Let's have a couple of laughs. In fact, let's have a billion laughs. This is the XML variant of an attack called the billion laughs attack. And what this attack essentially does It takes advantage of, if you'll excuse the phrase, XML being too smart for its own good. XML isn't just about angular brackets, it's a really smart format with a lot of functionality. One of the things that allows you to do is define entities, and even worse, it allows you to reference these entities by other entities. So this really small, textually wise, really small XML has a content of the lol9 entity. But if we look at the definition, lol9 is made out of, made up, excuse me, out of 10 lol8 elements. lol8 is made up of 10 lol7 elements. And so on and so forth till you go back to the 10 by the power of 10 elements. And it's really hard. It's really not hard to imagine that with a textually really small XML, you can expand this to something really large in memory. So once again, I'm not a DOS geek. I'm a DOS nerd. So I graphed this out, and in this graph, I have two, two lines, the size of the XML as a function of the number of LUL levels, and the size this expands to in memory as a function of the number of LUL levels. And we can see the discrepancy here really quickly. Okay. Up to four low levels. It's really small. Nobody really cares. Six, it starts kicking up. With seven levels, you still have a tiny, teeny, tiny textual XML. It's just 650 bytes. It doesn't, not even a kilobyte. But this expands in memory to about 30 megs. If you continue this graph in your imagination, which I didn't do because it would just be impossible to see, eight levels kicks up to about a gig, nine levels kicks up to something ridiculously large with a really small payload. So what can we do? Again, let's start with obvious solutions. If you don't use XML, you are not vulnerable to DOS attacks that rely on XML expansion. If you can do this, great. If you're designing a system from scratch, just decide that you will not use XMLs and you're fine. Quite often can't do this because your system or service is part of something larger, some whole flow, a new microservice in an Existing mesh of services, and the XML transport is predetermined for you. Similarly, even if you absolutely have to parse XMLs because that's how your configuration is built, that doesn't mean you need to allow XML parsing from tainted, external, untrusted sources. Again, if possible. If an attacker can't inject an XML into the system, even though the system uses XMLs in other places, you're probably fine. But again, this is not always possible or plausible. If you can't do this, Learn about the parser you use, learn about the library you use. Quite often, you can configure this library and switch features on and off. For example, if you don't need the capability to define entities, And I must admit, I've been a software engineer and an engineering manager for about 24 years. I think I remember one case where I saw a system that really needed to define entities like that on the fly in XML. So if you don't need it, switch it off for the common libxml wrappers like libxml. js in our example. You can set no int to false or huge to false and you're good. Of course, you can always use a library that just doesn't support these features. Again, if you can, if you don't need these features, just do it. If you don't need the power to define entities, don't give your attackers this power. And of course, the cardinal law of application security. If you're getting any tainted input from an external source, input that you do not trust or should not trust, sanitize it. Go over it, make sure there's nothing funky or malicious there before you try parsing it. Not after, that would be too late, before you try passing it. And I know this is a recorded talk, but even if, even though it is, and even though I'm just sitting here in my study recording this with no audience, I can already hear the Snickers. of people writing more modern applications in modern formats who don't use xml and never used xml and never wish to use xml and think they are safe. you are safe, of course, from any attacks based on XML if you don't use XML, but I'm sorry to be the harbinger of bad news. If you are using YAML, you probably have the exact same problem. YAML also allows defining entities, again, parser, of course, but generally speaking, the format also allows defining entities. It allows recursively. referencing entities from other entities. And just as we have XML bombs, we have YAML bombs. Luckily, the same tactics we discussed to prevent DOS by XML or DOS, or excuse me, XML bombs will of course work for YAMLs. But okay, enough about annoying formats that we all love to hate or hate to love, your dealer's choice. Let's talk about something even more ubiquitous to JavaScript. Let's talk about regexes. I dare you to write any good sized application without a regex somewhere. And if it isn't in your code, It's probably in some third party you're using. In fact, even this most basic Express application, even without the callback, has regexes baked in there. Because Express needs to take the URL, break it up, and understand what handler to match this request. So the regexes are there. But anyway, let's talk about, let's focus on the user code here. The user code. It's really simple. It takes two query parameters, regex and a text to test this, to test by this regex. Performs this test and returns whether the text matches or did not match the regex. Sounds straightforward enough. The problem here is that, and this is a recurring theme, regexes can be too smart for their own good. Or more importantly, too smart for your own good. Javascript tricaxes, which are PCRAs, Perl Compliant Regular Expressions, have a feature called backtracking. You can define a capturing group, and then a wildcard, that applies to this capturing group. And this capturing group can have its own wildcard. So consider for example a regex like the regex in the comment here. Open, open parentheses, a plus, close parentheses, plus. Now this regex in layman terms means A series of one or more As, one or more times, which, excuse the language, is a really stupid way of saying a string made up of As. We could have defined this as just A But for the example's sake, we purposely define it like this. Now, think about the poor regex engine that needs to take a string and test with this regex. Consider your input is 10 characters. The poor regex engine will need to test if it's 1a followed by 9a's, if it's 2a's followed by 8a's, 2 groups of 1a's followed by a group of 8a's, 2 groups of 1a's. Followed by two groups of four A's and so on and so forth. It gets really ugly really quickly. How ugly? How quickly? Again, as I said, I am a DOS nerd. we graphed it out. In this benchmark, I took a string which is made up of a series of A's followed by a single B character. This is to prevent the Regex engine from failing fast and saying there's no possible combination. By doing this, I have forced the Regex engine to check all the permutations up to the length of the string. And I just tied this regex with increasingly long strings, so a followed by a b, two a's followed by a b, three a's followed by a b, and so on, and graphed out how slow this is. And again, I am, I want to emphasize, this Speed itself isn't the issue. the time itself isn't the issue. The issue is that evaluating Techn, RegX happens on the Event Loop. So however long or short list takes will type up the event loop and make you make your application at least unable to serve other requests. So how bad is it up to about. 28, 29, let's say, for argument's sake, even 30, 31 A's followed by a B, it's not too bad. under half a second, we can live with that. And of course not. 50, 000 milliseconds is 50 seconds, but under 50 seconds, under a minute, we can probably live with that. But at about the 31 A's mark, it really kicks up. In 33 seconds, it takes 50 milliseconds, which is almost a minute. Okay. With 35 As, it takes a bit shy of 300. milliseconds, which is 300 minutes, 300 seconds, which is about five minutes. As you can see, this presentation has lost my ability to do math. So I will rephrase in 35 characters, we get about 300 seconds, which is under five, but under 300 seconds, which is a bit under five minutes, which is It's slow and annoying and let's say it's the outer edge of what you'd be able to expect when I'm trying to go up to 36 A's. This simple Node. js application just took up so much memory, so much CPU power, it completely killed my computer to the point where the music I was listening to while making this graph started lagging. So I just killed the benchmark and said, okay, that's it. And if you think about it, An attacker being able to send a payload that is 35 or 36 characters long has not that far fetched. It is not a big payload. This is a really easy attack if your application is vulnerable to it. So what can we do to prevent this? First of all, if you have a regex, baked into your application. Just check your regex. Make double sure, triple sure, quadruple sure that this regex does not have a backtracking part in it. There's tons of tools to do this. Generally speaking, SAST, static analysis security testing tools, are really good at this. Test your regexes. make sure that you don't have something super dangerous baked into your application. Now, if this isn't your regex, which is hardcoded coded in your code, if it's some valuable input, first of all, If you can, don't allow tainted input, user controlled input, as a regex. This isn't always possible, especially if you think about applications that have search capabilities with wildcards. It isn't always possible, but if you are going that way, maybe don't allow the end user to input a straight up regex, but have some a simplified wildcard syntax like allow asterisks and that's about it where you take the string and convert it to a regex in the back end in your code. So clearly it's still affected by user input but you can't say it's completely controlled by it and if a user can't input any arbitrary regex with backtracking It could be okay. Kind of following up on that, if you to have a regex. Maybe don't allow a regex that you know can be dodgy to evaluate user input, tainted input. Okay, this is usually not possible. We usually don't use regexes to evaluate data we know, but data we don't know, which usually comes from some external source. But if you are doing this, and you can't avoid having a tainted input, dodgy regex, at least, again, use length limits. We've seen in the graph that the time we tie up a regex is really dependent on the length of the input. So even if it is a really bad regex, but you limit the length of the input, a user, or in our case, an attacker can fail it on this regex, it could be okay. And of course, as always, maybe don't use regex, maybe use some alternatives. First of all, still in the Regex realm, instead of using JavaScripts built in Regex, you can use a package like RE2. RE2 is not a PCRE. It is not a Perl compliant regular expression. It is much, much less powerful. But, if you don't need all of this power, and you can suffice with the functionality that RE2 gives you, RE2, because it doesn't have all the bells and whistles, because it doesn't have bell tracking, backtracking is not vulnerable to Redos. So maybe that's a valid alternative in your use case. Also, maybe don't invent the wheel. A lot of the times you don't need a general purpose regex. A lot of the time you use regex for really scoped tasks. Like checking input, if something is a valid email, valid address, valid URL, then write your own regexes. Use proven tools that someone else wrote and tested and verified are safe. Validator. js, for example, in the Node. js ecosystem, has a ton of regexes, a ton of regexes. validators. Most of them are implemented by regexes to check inputs for a ton of needs, including All sorts of ISO standards, I admit, I never heard about, like standard for shipping container identifiers. If you can avoid writing your own regex, by all means do Now, the last type of DOS attack is something near and dear to my heart. Having done storage professionally for a lot of years before moving to application security. this again is a really simple application. Once It gets a request, it reads the file from disk. There's no user input involved here. Reads the file from disk and returns it back to the response. Now, of course, this is a really stupid application. If this were a real use case, I'd read this file once, store it in memory, cache it. But for the demonstration's sake, This is a valid way to implement such an application. And if this weren't a demo, you can think about an application that has various files to read or whatever. Now, the Tosya is really sneaky. Who's If we think about storage in no js, there are two ways we can do storage operations. We have the Async way, such as FS three deal, or Fs right file, which delegates the IO to a worker thread and all sorts of third party libraries like F. S, extra Graceful Affairs, A DMZ also have the same patterns, perform a call. You provide a callback, the IER happens. In a worker thread, in the background, does not block the event loop, and once the IR is done, you get a callback with the result. Now, if the first way of handling storage is the async way, the second way is clearly the wrong way. So any API and specifically storage that declares that's a sync API, so FS read deal sync, FS write file sync, same goes for ADM zip APIs, for FS extra APIs, for any API, frankly, not even necessarily storage related. is explicitly telling you that it will be performing its work on the event loop. Now, it's often more convenient to code like this because you aren't messing around with callbacks or promises and everything is just straightforward. But the price you're paying is that you are blocking the event loop. Now, this Sometimes can be okay, especially in the context of the application starting up and it's not serving user requests, etc. But if you are in the context of a user request, any API that declares that it's synchronous and blocking the event loop, should simply be out of the question. Should be something that you don't use and don't even consider using. Now, this form of DOS is sneaky because we have become very accustomed to personal computers, which have SSD and NVMe and all manner of really fast drives, that we don't always have this in the forefront of our mind. But storage can be slow, and especially outside of our development environment, while we deploy our code to some third party cloud provider, which uses some third party storage, which we have no control over and no idea how fast or more often than not slow it is, don't trust it. Don't give it the chance to block your event loop. Delegate everything to async APIs. And yes, it is a bit more complicated. It does arguably make the code a bit uglier. It does arguably make it a bit harder to debug and maintain. But from a DOS perspective, this isn't just a better way of writing code. It's the right way of writing code. Don't make your users suffer because As a developer, you were a tad lazy and wanted to just call a sync API and be done with it. Put in the extra couple of minutes. It usually isn't much more than that. Couple of minutes of effort and do your storage calls or do whatever call. If there's a sync and async variant of the API do it with the async variant. Your users will thank you with this, I think, Nearing the end. So just want to sum up here and give you some takeaways from this talk when we're thinking about application security, especially if we are web developers or application developers that don't usually focus on application security, we focus on our applications. of course, we want them to be secure, but we focus. first and foremost on functionality. We usually think about instinctively about injection attacks, about our SQL injections and XSS malicious payloads, doing malicious things to applications, which is all true. And if you're thinking about those and defending against those, That's great. But don't forget about DOS. Don't forget about denial of service. Quite often, from an attacker's perspective, it is considerably easier to exploit DOS vulnerability than an injection vulnerability. It takes considerably less skill to do so successfully. And while it might not be possible to steal all your data by DOS vulnerability, it is relatively easy to Bring your application to a screeching halt where it cannot serve its legitimate users. So you may have not lost anything, nothing was stolen, but your business isn't working anymore. How do we defend against DOS applications? There's a couple of really simple takeaways to remember. First of all, if we already are thinking about security, we do not allow user inputs or tainted inputs to places where they have no business to be. If we do, we sanitize them, specifically in the DOS perspective, we sanitize them against attacks that might exploit our parser, being XML, YAML, whatever. Regardless of sanitation, we use length limits and size limits. These are really easy to implement. They won't prevent 100 percent of the attacks, but they do make DOS attacks considerably more difficult to pull off. Also, even regardless of DOS attacks, mistakes happen, users do crazy stuff, don't allow them to bombard you with huge inputs if there's no business value for it. And lastly, think about the libraries, think about the APIs you're using. Think about whether they're running in the event loop or in a worker thread. And if they are running in an event loop, in the event loop, chances are there's either a different library or a different API within that library you can use to delegate to a worker thread. and learn the configurations quite often really easy to set up an application that would just work with the default configurations. Quite often these configurations aren't only not geared towards security, they aren't geared towards production usage in general. So once you have your prototype working, take a minute to review the configuration. Maybe you want to switch off feature flags. Maybe you want to add a length limit. Maybe you want to add some built in sanitation. Get to know your project. The APIs and libraries you're using, it won't only make you more security aware, it will make you a better programmer and it will make your application better. With that, I am really done. If any of this resonates with you, I do invite you to keep in touch. I am not a hard person to find. Can reach me on LinkedIn, on X, Twitter, by email, by all means, reach out. Let's keep this conversation going. And thank you. Thank you for listening. Thank you for your time. Thank you for considering the security of your applications and not just the functionality. And have a great conference.
...

Allon Mureinik

Senior Engineering Manager @ Black Duck

Allon Mureinik's LinkedIn account Allon Mureinik's twitter account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)