Hi, I am Tanya ka. Welcome to my talk at Conf 42. Instant response for developers and DevOps folks, too. Actually, this is instant response for literally everyone. So I do all sorts of work now. I used to do appsec full time, and I would respond to a lot of security incidents. And I don't know how to say this nicely, but sometimes people would try to help, and then they would break things and mess things up and ruin evidence and make my life really way harder than it needs to be as a security incident, either manager or investigator. And I was just like, oh, my gosh, why would they do this? So one day I gave training to the whole it department with my colleagues. Just to be clear, I didn't do this single handedly. So we taught help desk, we taught the system administrators, we taught the software developers. We're like, this is what an incident looks like, and this is when we need you to call us. And here's why we need you to call us. And my life got infinitely better. We spent less money on responding to incidents. We responded smoother. Our reputation didn't get as damaged. We were able to stop some problems from happening, and we ended up having better relations with other teams. And so since then, whenever I train software developers, because that's what I do, I chain people in secure coding and appsec, stuff like that, and I run community events. But anyway, whenever I train a team, I show them that. And basically, security incidents go better from then on. And security incidents are the most expensive way to deal with a vulnerability. It's the most humiliating way, time consuming way, scary and humiliating way. And so I want all of us to have fewer security incidents and the ones we have, I want to go better. And so that's why I have this training I do, or whatever. This is me condensing it down as fast as I can to help. You look like a superstar at work when there is an emergency. Okay, so who am I? I'm Tanya Jenka. I started this little company called we hack Purple, which was acquired by Semgrap this summer. And now I'm head of education and community, and I just do lots of training and free, fun community events, which is basically, like, stuff that I really like doing. So I'm known as she hacks purple. And yes, I have some purple hiding in my hair. I wrote a book called Alison Bob Learn Application Security, and I'm way behind on my next book, don't tell my publisher. So this is year 28 for me of working in tech. I'm an advisor at some startups I love OASP. I blog and stream and do stuff like on the Internet all the time to share information. But the key takeaway is that I'm a nerd at large on the Internet and I really care about the security of software. Those are the key takeaways and I create a lot of content. So enough about me, let's talk about incidents. So what even is incident response? What even is it? It's an organized approach of addressing and managing the aftermath of a security breach or an IT incident that has to do with something to do with security. So the goal is to handle the situation in a way that limits the damage and reduces the recovery time and costs. So I want to save money, I want to save time, I want to save reputation, I want to save employees from having lots of new gray hair. I'm so not kidding. Okay, so what's an event versus an incident? So there's actually lots of events all the time, but it's not necessarily a security incident. So on the right I like to think of that as an event, and on the left I like to think of that as an incident. Like she's for sure something bad has happened. Okay, so in simple terms, a security event is when something strange happened or is happening and you're suspecting something's wrong, but you're not quite sure and you need to do triage at this point. But a security incident is when you are certain that something bad has or is happening. So for example, if you find your data for sale on the dark web and you have this giant magazine write you and say, hey, Tanya, will you give me a quote about how you feel that your data is for sale for only $50? That is a security incident for sure. Okay, so this is my first talk ever with a trigger warning. I'm going to tell one story, so I'm going to have many, many stories throughout this to help illustrate my point. But one story is something where something bad happens to children. It's not graphic. I don't get into detail, but if you're a really sensitive person, you might want to step out. Security incidents can be scary. And I tell the story to explain the gravity of a situation. And again, no details, but I would rather, you know and leave rather than stay, not be aware and be upset later. So judge for yourself. Okay, so your needs, its software to be secure. It needs all your applications, your APIs, your SaaS products, your crops products. Like any piece of software, it needs to be secure. It does. Developers are our first line of defense in that matter. They just are. All of it helps it. Security works really hard on it. But developers, we're counting on you. And so without your buy in, us security folks are lost. And so if something happens, sometimes we need your help and we are not trying to make you do our jobs. It's stuff that we can't do without your assistance. And so there are five things that we need from you, and most of them are things we just need you to know. So we're going to talk about your role during a security incident. We're going to talk about need to know and a couple of stories. And the purpose of this is so that you know what to do and why. So you're not just blindly doing stuff. You understand the value. So the first thing is, if the security, so tell the security team, if you see something, it's better to report something and have it be nothing chain have it the other way around and not report it. I remember someone calling me once and he was like, oh my gosh. All the super sensitive USB drives, because this was back in the day just are missing from my desk. Oh, my gosh, there's the sensitive information. I had it, my drawer locked. I had my office door locked. I don't know how this happened. And he was just freaking out. And I was like, okay, one, go outside your office, ask if anyone saw something. And he's like, okay. And he goes out and then his administrative assistant was there and she's like, oh, hi. The courier came up, the super secure courier came and he picked up the drives. I let him in. I hope you don't mind because you were at lunch. And so he had come back and she'd been in the bathroom, she'd missed him, and he'd been freaking out in his office. And then he said, oh, my gosh, you must think I'm an idiot. I'm like, no, you followed the policy literally. Exactly. You're a hero if this is not what had happened. And you'd run around fooling around for an hour or two. That's an hour or two where the criminals gang away with our stuff, right? So you followed the policy perfectly. She should have left you a note and told you. She assumed she would have seen you, right? Like our fingers got cropped, but you took like, what, two minutes of my time? You're great. Thank you for following the policy. And he's like, oh, I'm like, I'd rather a false positive anytime over a false negative. Some things that you might see would be like, what if your APIs keep going down. They keep crashing all the time. Or there's a certain API that just keeps crashing and you don't know why. The monitoring is like, oh, it's so heavy. There's so much latency. It's so slow, and you look and you can't figure it out. Call the security team. There's probably a bot or some sort of brute force attack happening or something. Let us help. If there's something that you're like, that's just so weird, why is this happening? Call us. We don't always know the answer, but we can tell you if it's a security thing or not. Usually pretty fast, and sometimes it's that our tools aren't configured quite the right way or maybe forgot to. Tell us about this new API you launched recently. There's lots of reasons we might not have seen it, but if you're like, I just can't figure this out, call us. We want to help. Okay, so that's, one, if you see something, please say something. Two, if we call you into a security incident, please don't leave for the day without telling us. So I was responding to a security incident this dev had cross site scripting in his app. He had all sorts of other problems, too, and someone was actively attacking it, like, actively exploiting his app on the spot. And I was like, okay, we got to do this. We got to do that. We need to release an update. And he's like, okay. I said, listen, I'll be back in a few minutes. I just have to go brief my boss. We finally figured out what's going on and which app it is and who to talk to, and he's like, okay. And then my boss and my boss's boss and my boss's boss felt the need to panic a little bit, because that's what happens sometimes, and it's my job to cool them down. Like, calm down, calm down. It's going to be okay. So I ended up taking longer. I thought it would be 1015 minutes. It was like 45 minutes. I come back and he's gone. And I say to the cubicle mate, hey, where'd he go? And they're like, oh, he had to go catch his bus for the day, so he left. He's like, what? Do you have his number? No. Do we have, like, a pager? Because chain, I'm old, and pagers were a thing. And they're like, no. And we don't have him on call, and we don't have the right, according to the union, to phone him unless he's on call. And also, we just don't have his number because we never put him on call. So what do you need? I'm like, I need to release a bug fix on this app, like, right now. Can you push this code? And he's like, no. I'm like, I can fix the code. Can someone push it? And no one was there that had the access and permissions to push the code, and it just got hacked all night. And I'm sitting there powerless. And I didn't have a waf in front of it. I didn't have a content delivery network. I couldn't deploy virtual. I couldn't even turn it off. We just had to sit there. Eventually, I went home and I was just like, so please don't leave. If you have been brought into an incident, unless you tell them, so say, like, hey, I'm leaving. Here's my number if you need me, because I could have fixed that bug. I did. And the next morning, I left all these instructions for him. I'm like, oh, he's things immediately. And apparently the attacker stopped partway through the night because they were tired of just owning us. And anyway, and so he fixed that. We did tests. It was fine. And I was like, please don't leave without telling us. I can't do this without you, literally. Okay, so that's number two. Number three, this is an emergency. I have had devs just like, go to a regular team meeting, go out and have lunch, do all these things. I'm like, this is literally an emergency. That's why I called you in. If you're going to leave your desk and go do other things, or like, they're like, oh, I have this deadline on Friday, so I was thinking I'd work on your thing tomorrow. I'm like, I don't care about any of your deadlines. This takes top priority. Do you need an email from the CEO to tell you, like, stop doing that other crap? Nothing else is important except this right now. And if your boss questions it, send your boss to the IT security team and they will explain the level of importance. We don't crank you in unless it's really important. We do every possible thing we can ourselves without your help because that's our job. But if we call you in, it's because we absolutely can't do it without you. And so all your other work stops. It's only the security incident. We will talk to your boss, no problem. Just like, oh, my gosh, please treat it like the emergency that it is. So that's number three. Number four, follow. Need to know. So unless someone else needs to know a specific thing about the security incident, don't tell them. And that includes your boss. I have many times run interference on behalf of my investigators where the boss is blocking the cubicle, and they're like, you can't take my employee's computer because I trust him and I know him, and I promise he's dead. So you can just leave now. Get out of my way. I don't have time for your crap. You think you're being loyal to your employee or whatever, but we all work for the same organization, and if we spot some, like, we have the right to look at someone's computer. We only do it if there's a pretty darn good reason, and that doesn't mean your employee is bad. It could be that someone broke into your employee's computer. And I actually find that to be way more often the case or there's a misunderstanding, but your boss does not need to know. And so lots of times they'll say, hey, I need to brief you on the security incident. And I bring them into a room, and then I run little circles around them and give them no details and waste time while my team goes and removes the computer or asks the person for an interview or calls the police or whatever it is they need to do. I'm like, I'll go distract the boss, no problem. Your boss does not have the right to know. They get to know you're working on a security incident. They totally can go ask the security team for more details. But if they're like, hey, show me the code you're working on. No. Oh, what's happening? No. Okay, so first story. So I was late for work one day because I had a dentist appointment. I really like having nice teeth. I want to still have teeth when I'm old. So I go to the dentist, and everyone knew I was at the dentist. I show up at 1030 instead of at nine, and I walk towards my team, and they're like, meeting room, meeting room. Go. The big meeting room. Now there's a security incident. I'm like, but all of you are here. Who's managing the security incident? They're like a help desk guy. I'm like, help desk isn't on our team. They're like, go. They told us we're not allowed in. Like, what? So I go in. So this organization had two buildings. There is one main headquarters building where all of us were, and then there was a side, like, smaller building across town where maybe, like, 10% of our people worked. And basically, they were having Internet troubles. They couldn't get anything to load. Everything was not working. And so people had called help desk, and the help desk person decided he would be a hero today and that he would handle it himself. And so he started calling all the executives and telling them that our building that was made of concrete, it was not an Internet connected, smart building. It was a regular concrete building, was infected with malware, and that the whole building had malware, and it was not safe. I walk into this room, and all the sea level executives are losing their tops. They're totally freaking out. They're like, oh, my gosh, Bob. And they're like, where were you, Tanya? I'm like, at the dentist, just like my out of office said. And my team told you, and we needed you, and you weren't like, I had my phone. You could have called. Actually, you literally have my number inside each one of your passes, and I laminated it, so you had my number, and they're like, well, we needed you, and you weren't there. I'm like, so you didn't call me, and you're upset that you didn't call me, okay. And I'm like, so what's going on? And they're like, well, this building has malware. I'm like, that building is made of concrete. It does not have malware. And they're like, oh, you don't know. I hope he knows. And he's the boss. Okay, so here's what happened, everyone. So Canadians love the Winter Olympics. We love it, and we the most love hockey and ice skating. And guess what was on that day? Ice skating. And so in the canadian government, there's generally this don't ask, don't tell rule when it comes to watching the Olympics at work. And so what they do is they'll have one boardroom in each building, and they're like, the Olympics are going to be showing there. We know it's the Winter Olympics, and if you watch it from your desk, you're in trouble. But if you want to go see how whoever does at whatever go to this boardroom and they just stream it for that week or whatever, and we just give up the productivity hit, because otherwise people stream it at their desk, and it's always, you're not allowed streaming at your desk, because if every single person's streaming at their desk, there's no Internet. Right? So apparently, the head of this building decided to go against government policy and say, if you want time off to watch the Olympics? You're taking vacation days. I don't care. I am strict. I am a Grinch. You're not doing it. So every single one of them, like literally 80, 90% of employees were all streaming the figure skating. So there was no malware. However, guess what happened? All those executives. So what I do, as soon as I start an incident, I'm like, I am the incident manager. Here is what we're going to do. You are not going to tell people outside this room anything except that the security team has tapped you for an incident. That's all they get. They want more info, they come to me. It's called need to know. And I'm like, bark, bark, bark, bark, bark. Hope that guy didn't do that. So they all told their admins, their admins told all their friends. For months I would get in the elevator. And I remember the time where that building had malware. It was so scary. I wonder how many people were hurt. And so one, don't run around and tell everyone. Number four, don't run around and tell everyone. It is no one else's business, including your boss. I'm not kidding. And I know that this is a ridiculous story. It wasted thousands upon thousands upon thousands of dollars in people fretting and worrying. People in that building want to evacuate because of this help desk guy who didn't know what he was talking about. And he kept calling the executives and taking calls when I instructed them to talk to me, it took me three days to get full control of the incident and to get all the executives to listen to me again because they felt that I was being a bit of a sourpuss about it. And I was like, this is what has happened. This is what Wireshark says. This is what our network, this is what says. And they're just like, we don't believe you. We believe the help desk guy. I'm like, okay, so, number five, do not try to manage a security incident yourself and be a hero. And so the last one also illustrated this, but it was more about need to know and telling a whole bunch of people that shouldn't know and then literally terrifying 1000 people. So this is where the trigger warning comes in. If that is a thing you want to step out for, step out now. So not at my office, but the incident manager who first trained me, his previous office, one of the help desk guys, had decided he would manage an incident and he stumbled across images of child abuse and he accidentally viewed a bunch of things that he could not unsee and in doing so, then went and deleted a bunch of things because obviously it's offensive, right? And then freaked out and then did a bunch of things and ended up calling the security team. Well, what he did was he ruined the chain of custody, which meant none of it counted as evidence in a legal way anymore. And the person who had been doing this very extraordinarily terrible thing got to go free. And that employee ended up in therapy for the person I worked with. He's like, after four years, he was still in regular therapy over what he saw and could not unsee. And because of his part in essentially letting this criminal go free. And this was something where they'd go to jail for the rest of their life. And there are all sorts of other people involved and they could have uncovered things, whole ring of people. If instead when he stumbled upon something, if he had immediately called us, we could have taken care of it. When you become an incident responder, you are accepting a certain amount of risk. You understand you might be exposed to things that are terrible. I did counterterrorism for a while and I had nightmares all the time. And I have a lot of respect for people who are able to do that work longer than I did it. I could only do it a year and a half. And I was like, my mental health can't handle this anymore. I can't imagine things like that. They were outside of my scope of imagination, of awfulness. And so as a result, I do softer, gentler, absec work. Now, when you are an instant responder, we often know when not to look. We know how to manage the evidence so that we don't ruin it. We know what tools to use to collect things in a certain way. And we know when to call the police. And sometimes you just call the police right away. And the thing you do is don't touch. And so we agree to carry that burden. When we do things work, we agree to it. You didn't agree to this. This isn't your burden to carry. So if you see something, just call us and we will come and help. It's not your job. You should not have to do this. You should develop awesome software. You should patch servers and harden them and make sure they're safe. You should monitor and log and check that everything is wonderful and that our products delight our customers. That's your job. This stuff is for us. And we've agreed to these terms when we accepted this position and you didn't. So please don't try to be a hero. I know that you feel like you're helping, but more often than not you're actually hurting. And sometimes the hurt is really serious. And so with that, I wanted to just have this here for you so that you have a second so that you can take a look at that. So I am putting this picture here for you to take a screenshot if you want to. These are the five things that we need you to know as incident responders. Tell us if you see something. Don't leave the premises without telling us. If you are part of an active incident, this is the top priority for you. If you are part of the incident, if we've asked you for your help, please put everything else down. Follow need to know don't tell people that do not need to know. It doesn't matter if they want to know. I don't care what they want. Number four, do not try to manage it yourself and be a hero. Please call us and we will take this burden from you and we will handle it well. And with that, whatever it happens with your software. If you are not sure, ask the Appsec team. If they don't know at work. We hack purple community can help you, or Samgrep community. So Samgrep bought Wehack Purple community. And so you might be by the time, if you're watching this, after the live conference in the coming weeks, we are moving to a new platform, but basically join we hack Purple, the Samgrep community and attend way more stuff like this. It is the security team's job to help you, right? So most of the time they can answer your question. If they can't, we hack Purple can help. It sounds weird, but I see them all helping each other all the time, all the community members, and there's 8000 of us. So there's a lot of nerdy help there. And with this, I'd like to lighten things up a bit and give you some resources. So the first resource are a whole bunch of books about DevOps and my book about appsec. So I feel we can't do security right if we're not doing it right. And I feel that the best way to build software is by following the DevOps principles, by using the awesome kick ass tools, the awesome processes, the really great products. I am a big fan of DevOps. So those first four books are about developers and all of them are awesome. And there's a new one called investments unlimited and it's really, really great. And so I suggest that one too. And of course Alice and Bob learn Appsec. Me and my mom agree best book ever. Every single Monday on Twitter, on Blue sky, and on infosec exchange server for Mastodon, I run a mentoring program. So I don't pair people individually. People ask for a mentor and then other people swoop in and answer them and help them. Sometimes it's just a virtual coffee or a recommendation of a book or join this local community or come to this event with me. But sometimes it's a friendship that lasts for years, and I've been doing it since 2018, and thousands of people have met each other and formed friendships, formed companies, hired each other, taught each other, and grown and connected together. And so I invite you to take advantage of this small, tiny program that I do to try to help the community. We hack purple community. Please come join us. It's free. All the courses inside are free. All the events that we hack purple. So sometimes we advertise external events that we like, like the Diana initiative or Cyberjitsucon, et cetera. But all the events by Wehag Purple and Sungrap are free. And there's so many awesome humans and fun things that we do together and share. So that is one place. And then we are eventually moving over to somegraph community. So we don't have a platform yet, but we do have a newsletter which will keep you up to date on literally everything. And we have a slack channel for now where we can get you all set up with various things and then if you just want to try it, you could just play with the product because they have a free version. But basically, if you join the newsletter, you will be invited to all of the cool events that we do do every single month, which includes free trainings, workshops, talks. We're hoping to arrange a tabletop exercise for early next year and basically I get to have fun full time and it's pretty great. Lastly, resources me. So I have a website, I have a blog, I have a YouTube, I have all the Twitters and the Instas and all the socials. If you see we hack purple, most of the time it's me. I do have an imitator now at this point, which is really awkward and weird. So I am not trying to sell you gambling, just to be clear, but every single other thing is me. So if you look up Shehex Purple, you will find lots of stuff, usually about the security of software. And with that, I want to say thank you so much. Thank you for one, being here and watching my talk. There's a lot of other things on the Internet you could be doing. There's probably several cute cat videos you haven't seen yet, but you chose me instead. So thank you. Thank you to comp 42 for having me again. I really appreciate you. And I appreciate you making this really cool conference all about incident response so I can learn more from my peers. And thank you not only for watching, but for doing the work that you do. Because together we make it a thing that fascinates, entertains, takes care of, and serves the entire world. Pretty much almost every single person in the world is positively affected by what our industry does. So thank you for your work, and thank you for trying to report security incidents from now on, as soon as you see them. I'm Tanya Janca and I will see you next time.