Hi, I'm Elon Finkelstein. I'm Peter Van Eyfren. And today we're going to talk about the CISO is dead. Now what? Building quality DR, IR, and BCP exercises and simulations. Very quickly, as mentioned, I'm Elon Finkelstein. I'm a principal security architect on the security development team at OWN by Salesforce. I can be reached at elon. finkelstein at owndata. com. I'm also available on LinkedIn. I currently have a master's in cybersecurity from NYU, a bachelor's of science in computer science from John Jay, and I'm multiple times AWS certified and a member of ISC squared. Hi, I'm Peter Van Iperen. As I mentioned, I am the chief information I'm a former Deputy CISO at Clear, Fox, and Disney. I also have a Master's in Cybersecurity. I'm a former Deputy CISO at Clear, Fox, and Disney. I'm a former Deputy CISO at Clear, Fox, and Disney. I also have a Master's in Cybersecurity. And I have, 25, more than 25 cyber certifications. I also, teach some cybersecurity courses. Okay, so very quickly, we're going to just talk about our overall topics for today's talk. we're going to walk through an introduction, a quick review of the terminology, developing real world possibilities, introducing randomness into exercises, conceptualizing Black Swan events, gaining management buy in, including others. And then some follow up and conclusion points. So very quickly into the terminology, in order to build a common body of knowledge, disaster recovery or DR is an organization's ability to restore access and functionality to IT infrastructure after a disaster event, whether natural or caused by human action. I think what's more important here is not what this definition says, but what it doesn't say, which is, it's restore access and functionality to IT infrastructure, but really it's. What keeps you operating? Second would be incident response. Incident response is the actions that an organization takes when it believes IT systems or data may have been breached. security, for instance, security professionals will act if they see evidence of an unauthorized user, malware, or failure of security measures. That's from Microsoft. BCP, or business continuity planning, that's documentation of a predetermined set of instructions or procedures that describe how an organization's mission slash business principles will be sustained during and after a significant disruption. That's from NIST. And finally, a black swan event. A black swan event is an event that defies expectations. They carry extreme consequences and are only understood in hindsight. That Forbes. I think a good example is. COVID. COVID was the black swan event of the last 10 years, right? No one saw it coming, totally unexpected, shut down almost everything. And I, I think another type of black swan that we have to look at here very seriously in this is, not throw anyone under the bus, but CrowdStrike. CrowdStrike, what happened inside CrowdStrike, probably preventable, predictable. the effects that occurred to the rest of the world. Not so much. and I think that we need to really look at, true code, black, and then, and events that, start out as something predictable, small problems that, aren't taken care of in places, and they build into something that has truly unpredictable consequences. Yeah, absolutely. Absolutely. So in developing exercises, we want to build quality simulations that rely on a whole bunch of different factors. some of the ones that we want to talk about are clear purpose, well defined, authentic, interesting, exciting, unpredictable, inclusive, and automated. clear purpose, successful exercises begin with a straightforward and simple explanation of the goals. That's really just basically laying out exactly what the idea is, what our plan to do here is and what the end result will be. and the more clarity we can provide our participants, the easier and much more fun that these exercises will be. And I think we should note here that when we say Expectation of the end goals. It's what is the outcome? What are we more prepared for? What have we learned from this? Not this is how the beginning and end of this simulation occur, everyone. I think that's precisely what we're here to talk about not doing. So I just want to make sure we clarify that. Next up is well defined, right? We really want to have a good explanation of what is fair game, what's out of bounds, and what the parameters and boundaries of our exercise include, and also don't include. Generally speaking, don't include production. Don't include production. I'm going to say it one more time. Don't include production. Next, we want to make sure that our exercises are authentic. So they need to be built on real world possibilities and true potentialities for the organization. I would just add to that, that, true possibilities, true potentialities, flexible, and you're thinking of that. Precisely to the point of what we'll talk about later in this talk, black swans are unpredictable. there is a reason that the U. S. military has plans for, a zombie, apocalypse. It's not because they think a zombie apocalypse is going to happen, but it's because it illuminates all the different problems that, rear their heads when dealing with an issue like that. So I think, be flexible in the thinking of what is, included in your exercises. Better to be more prepared, for things that might be somewhat unreal. Then, not prepared for things that could be very real, a la COVID. Absolutely. Absolutely. Next thing we want to do is make our exercises interesting, right? We need to capture the audience, right? We need to make sure that our participants, are invested in the game. And we need to do that by developing enough of a challenge, right? But still something. Solvable, that folks are interested for long periods of time and don't give up in frustration, right? We don't want them slamming their hands down and saying, I'm done, but we also don't want them finishing in 35 minutes, right? And I also think that, solvable, achievable graspable is important. always ends in success. It is not important, and as a matter of fact, not ending in success is often more important than being successful in all of these exercises. I think the next part we want to talk about is the unpredictability, right? Exercises need to have enough randomness of events and outcomes, that keep the participants interested but also on their toes. So there is no expected outcome, just to what Peter was saying. We also want to make sure that our exercises and simulations are inclusive, right? We want to remove the typical cast of characters, right? Those who are constantly or regularly involved in incident response, disaster recovery, or BCP planning, right? We want to substitute in the folks who may have less experience, the people who potentially are the juniors on the team who have never been involved Incident Response, or BCP, or DR, right? That's what's really going to both cross functionality train our team, but also help to teach folks who haven't had a chance to lead an incident or an incident response how to do so effectively. They used to have a hard and fast rule, if you wrote the BCP or the DR plan, you're not in the exercise. And I think the final one is automation, right? So we want to use scripting and automation to develop scenarios that unfold without organizer interaction, right? You don't want to be leading one of these exercises and Be stuck behind a keyboard trying to manually trigger events Trying to generate some sort of lock that can be used for an alert creation or anything else, right? We want to be involved with the players themselves And we want to leave all of the preparation to an automated And I just want to be clear here, if anyone leaves this talk thinking, I need to go, to write simulated triggers through an AWS environment to do what we're talking about, that'd be fantastic. and I've seen it done and I've seen it done really well. Simply numbered and rolling base or, decisions that are made, sufficiently would bring in enough randomness that is out of your control and also leaves you from, disengaging with the exercise. Okay, so introducing randomness. when we develop these exercises, we want to make sure that we include a lacking or definite plan, right? to introduce exercise, to introduce randomness into our exercises, it serves really two purposes, right? We help to maintain the participant interest. It also makes our exercises really authentic, right? Because nothing that in a true BCP, DR, or IR. Our situation is going to be predictable, right? Everything is going to go haywire. Everything, might go out the window and you need to be prepared for that. And I think when we introduce randomness, we can do it in a lot of different ways, including rolling dice, including picking envelopes, including simulated random number generators and options. I think also something interesting to do is treat it like it's a real disaster. Take shifts. Make people step away to take food breaks, to take, go to the bathroom, be a human, get tired. That's what's going to happen in a real world scenario. We're not all going to have a neat hour and a half break where we all get to have a nice conversation in the middle of a disaster. so treat it like a real disaster. I think the second part, when it comes to introducing randomness, is it helps to maintain interest, right? when we're able to limit distractions and develop something that is really engaging and really interesting in, to our participants, right? We can keep them involved and we can also make sure that there's no predictability. We're able to remove any expected outcome and make the game very interesting and unpredictable. And I think it's important to dedicate the team to doing that and taking away distractions. And I think it's really important too that whoever is running the simulation doesn't know when the simulation is ending or how it's going to end. That is. That is a disaster or a business continuity recovery, right? You don't know when it's going to be over. And I think that helps keep people engaged because if you tell people, Hey, we could be in this room, it could be an hour, it could be 8 hours, it could be 12 hours. We're all going to dedicate ourselves to really simulating this. You really start to understand what will occur in a disaster in your teams. And, it also focuses people, right? just generally speaking. in the first half hour, people want to check slack after they've been at it for a while. If they want to avoid the game and they want to get out, I think distractions will go away very quickly. Very good point. Very good point, Peter. And next up, we'll talk about some Black Swan events, right? So it's really hard to figure out the one in a million possibilities, right? If they were well known, they wouldn't be Black Swan events, right? I think a lot of what goes into this is the creativity and understanding of your actual environment and what would really derail it. I think that, how do you stimulate a black swan event, right? How do you simulate the unknowable? I think if you, research and look at people who study black swan events, one of the characteristics is that they're always. I don't want to say predictable, but knowable in hindsight. The reason they're knowable in hindsight is you can see the seven or eight factors that interacted with each other to create that black swan event. so if you want to simulate black swan events, create seven or eight random factors. Let them interact with each other and see where things lead. You may not end up in a black swan scenario. You may end up in a very vanilla scenario that, that is. a typical, DR scenario, someone kicked the court out, right? but you also may end up with a truly unique scenario to you that really starts to highlight how you would have to act during a black swan event. definitely. Now. Okay. Gaining management buy in. So convincing management that half the ops team, three quarters of our security resources and legal finance and product all need to be in the same room for the next eight hours and not at their desks is not easy, right? it's only a few people from the company. who needs three quarters of the company working? It's going to be an uphill battle. I don't have a lot to add here. I, in security, we're often. What is the ROI of something not happening? What is the ROI of things being proven negative? Ah, what's the ROI of something that not happening? and preventing the bad thing from happening. but I do think, Ilan, you have some good points, pointers here on how we can help people make a case. So absolutely. So really quickly, we need to have a good understanding of what motivates business, right? Typically, business is motivated by So I think in this case, I think it's just about the ability to supply the ability to continue to be in business, right? Our ability to be successful. and some of that comes down to our ability to make sure that our systems are continuous, that we're, our data is not leaked. in addition to being able to make money, right? So I think if we're able to show the value add that a lot of DR, BCP or IR exercises can bring to the organization, we're able to see much more buy in from our leadership. to really help make this point, we have some really good stats here that hopefully can help as you translate them to your business leaders. So very quickly in terms of BCP, continuity breaks cost somewhere between 137 to 16, 000 per minute, right? Nine out of 10 businesses experience one or more continuity breaks in a quarter, right? And I think this is just a really important thing to hit home on here. I think often when people try to make these cases, they try to say, the average cost of a server going out. we have that here, but really the most important thing in this slide is 9 businesses experience one or more outages per quarter. Go find that last outage, go talk to the product people, go talk to the sales people. Go find out how much that outage cost for your business. And look at the time that it was out and then go make that multiplication. It's the multiplication. That loss of revenue or that refund to a customer, et cetera, is a much healthier and more significant argument than how much it costs to fix a server. Absolutely. Absolutely. So some really quick DR stats, right? Corporations lose 40 percent of employee productivity during outages, right? 30 percent of organizations permanently lose data and more than two Thirds of how do I do that? Where you see all these big companies that have large outages cost companies over a hundred thousand dollars in revenue. Sorry, go ahead. No. taking this all back to your leadership helps to make that case. And I think just again, focusing in this slide here on people know what it means if 40 percent of people can't work, especially recently people live through COVID. They understand that. I think the other really important one on here is 30 percent of organizations permanently lose data. In your scenario, I am. Encourage everyone in your scenario, somehow lose data, somehow have data inaccessible, even if it's through randomness and it's how much of the data or et cetera, but have them feel the effect of not having data at their fingertips. Businesses operate on data and the only thing scarier to lose in revenue is data. 'cause data loss always leads to revenue loss 100% of the time. It leads to revenue loss in one way or another. Really good point. Really good point. And I think the third one is probably one of the easiest cases to make, and that's the IR stats, right? IR is probably the most well known of all the things we're talking about today. But again, these stats are overwhelming when we present them to leadership, right? Ransomware gangs collected 1. 1 billion in 2023 from their victims, right? 72. 7 percent of organizations worldwide have experienced ransomware attacks in 2023 alone, right? And cyber Cyberattacks exposed health records of 25%, 1 in 4 people, in the first 9 months alone in 2023. So the idea that you're not going to start to prepare for these basically eventualities seems silly. And I think just adding to that too, the ransomware and attack landscape is shifting significantly. You're seeing more places where data will just be exposed, data will not be returned. I, Brandon. It's where gangs don't want to negotiate, insurance companies don't want to pay. So again, adding that randomness, adding that fluidity into your simulations is more important than ever, because when you are faced with something like this, which 73 percent of us were faced with in one year alone, you don't know what you're going to be facing. and it's something at, in the security industry. community we talk about all the time, but other business leaders don't realize that and assume that there's one or two really easy ways out of these scenarios. And I think it's really important for them to understand that's not often the case. absolutely. Okay, I think we're coming into another really important part of a successful exercise, and that's including others. So when we talk about including others, once we have management buy in, right? We want other teams to participate, right? When we operate in, let's say, DR, BCP, or IR world, right? We're not operating with only technical folks, right? We need to make sure that all of the teams involved in an incident, right? Any of those types of incidents will be prepared, right? And by doing so, We can ensure that, again, we have a better incident response or IR or DR or BCP. So I think if you're performing a simulation and a lot, we're looking at the list of teams here, and I want you to tell us why it's important, but I just want to say to everyone, if you're doing a simulation and it doesn't involve at least two of these other groups on here, you shouldn't be doing the simulation because you're not simulating something serious. You can have a big enough that it actually matters to simulate it. You're not rehearsing something that is actually, significantly damaging to the company and that's what we need to prepare for. many of the little things we can handle, it's the big things where we need cooperation and we need to know that things will continue operating. Go ahead. Absolutely. And I think some of those teams that operate during, whether it's a DR, RBCP or IR in real life world. include finance, legal, our C suites, our physical security teams, and our marketing and PR teams, right? Because as technologists, we are only privy to a small scope of the business. These folks handle the other important pieces of the business. And I think that, going back to the cogent title of this talk, the CISO is dead. which is something we talk about internally here. I like to think I won the lottery, but I'm fairly certain a lawn kills me in a plane crash. regardless, I think it's important, especially across the C suite, to do that, right? Often in these plans you have that legal, or privacy, or your chief legal counsel or someone like that will sign off on the language that's released, or, your internal comms, or something like that. we often don't look at who the backups are for these people. And we They often take for granted that these people will be available, and the first thing that will occur in a real world scenario, in a real disaster, especially a large scale black swan event, is some of these people are just not going to be there. They're not going to be available. And the last thing you want to do is not know who. If you don't know who, you're paralyzed. And that is literally the number one thing. People can do things. If people, but what's the point? It's the old saying a long, a person is smart, people are dumb, a crowd is dangerous, right? People in a group, a crowd of people getting on a phone, no one knows who's supposed to be doing what, who's in control. Things are going to go sideways really quickly, so I think that's really important to bring into your scenarios. And you get to kill me in a plane crash, so who doesn't want to, take their boss out for a day? I guess now we can get into our conclusion. I guess the first conclusion The final conclusion from this talk is the CISO is not dead. He's very much alive and here with us. So that's a plus. but also, it's developing really interesting and seemingly random exercises requires time and attention to detail. it requires management buy in and also a level of authenticity, while being anything but average, right? So like we talked about, you want to include some of those black swan type events. You want We want to make sure that you're not utilizing the same resources that are utilized every time a real incident occurs, right? We want to make sure that none of the incidents or simulations that we're developing fit into any of our established playbooks, and we want to make sure that we really keep things interesting for our participants. And inclusivity helps a lot with that. I just to take the temperature really quick here too, I would just say that if you run simulations once or twice a year, and When everyone walks away from that simulation feeling really good about themselves, you've done it wrong. Just hands down, you've done it wrong. I don't care if you do it twice a year, every year for the past ten years, if you walk away feeling good, you're not doing it right. The whole point of doing these simulations is to be over prepared. You should be losing, you should be noticing things that are wrong, and you should be walking away with kind of an ick in your gut. oh, this isn't, this is, we didn't account for this. If you're not doing that, and that's at least not happening at least the C suite level, you're not doing something right, and you need to reevaluate how you're handling your simulations. Absolutely. Absolutely. I guess the next piece to just follow up on once more is that when we're developing these exercises, including members of departments that are peripheral and not necessarily technologists, right? Enhancing. It extends our overall unity. It helps to prepare us for when we are on calls with these folks, during real incidents or during real responses. And it also builds some sort of cohesion, That exists going into the next real live disaster recovery event. Or business continuity event or a real incident response. Oh, I, can I ask you a question? Sure. If we, if we kill a computer, does it complain? No. We talk about shooting computers like cattle, right? Yes. I'm fairly certain that computers are not what matters when something breaks. The customers matter. The clients matter. The workers matter. And I'm fairly certain that last time I checked, InfoSecurity doesn't manage all those departments, right? No. you're not having a real disaster if only InfoSecurity is involved in your simulation. Absolutely. Absolutely. And I guess the last couple of things is just to make sure that these exercises are fun and interesting, right? but also making sure that you have the organizational support, and getting an executive sponsor, along with providing that sponsor and your management clarity about what the exercise entails and what they can help to do to ensure success, right? Like we talked about in the management buy in really makes a big difference. Because at the end of the day we want to do this multiple times. We want to continue to practice, continue to build, continue to grow, and continue to improve. And the only way we can do that is by continuing to practice. And I think continuing to practice means that we need to take what we did wrong and evaluate it. In a way that isn't repercussion based, isn't consequence based, but is instead improvement based and say, okay, next time we're going to do this better and we're going to evaluate ourselves to make sure that we're going in the right direction. And I think that if we can show market improvement every time, it, you will gain that executive buy in and that sponsorship, right? There is at a certain level, everyone's job becomes basically care. That's what you do for a living. And, what keeps you up at night is the inability to take a problem that's gonna come along and solve it. having the confidence in the fact that You can have an outright disaster come along and you will at least be given the time and opportunity to figure out how to solve that through your disaster recovery, through your business continuity, through your incident response, and know that it will work when the big one comes and that you will not be S. O. L. Pardon my French. is something that most executives, when they understand the value of that, will 110 percent support, building out a program that does these simulations, and most importantly, building them out in a way that's random, and building them out in a way that one day, they will be the C suite officer and they Who is dead, and volunteering to do that, to see how their team responds and react. so a lot of final words. no, just to go out there and have fun and thank everyone so much. really appreciate you joining our talk. Peter's not dead, so if anyone comes and knocks on my door, I didn't kill him, it was someone else. However, do know, I do turn up dead. There is a gentleman we're watching right now, who we should talk to first, before anyone else. But seriously. It was great doing Elon, and I think that, This is something that is so overlooked in our arena. it's often a compliance checkbox and yet, it proves time and time again to be the most important thing, that we do, and, really appreciate you including me in this, even if, you had the threat to kill me. So thank you so much. And thank you guys for joining. I thank you. Bye.