The Charm of Incidents

Video size:

0:09 Miko Pawlikowski

Welcome to Conf42Cast, the tech podcast from a neighboring galaxy. My name is Miko Pawlikowski. Welcome to the internet, I'll be your guide. Today with me Emily Ruppe, solutions engineer at Jeli.io How're you doing, Emily?

0:23 Emily Ruppe

I'm doing great. How are you doing today?

0:24 Miko Pawlikowski

I'm great. I am wondering, if you could go faster than the speed of light, where would you go? That's literally what's on my mind right now.

0:34 Emily Ruppe

I would go to the Carina Nebula. It's one of the nebulas that the James Webb Telescope just took a photo of that they're calling a star factory. I want to know if it looks like that in real life. Or if they've just digitized it to make it look like cotton candy covered in stars.

0:52 Miko Pawlikowski

What if they did? What if it's black and white, would you be disappointed?

0:57 Emily Ruppe

I mean, that would be fascinating. I don't know if I would be disappointed or like privileged that I got to see it with my own eyes. And I could come back and report what nebulas actually looked like to the naked human eye. I don't know that I'd be mad. Because they don't know, right? They haven't seen it with their own eyes. it's kind of their best guess.

1:15 Miko Pawlikowski

Yeah, it reminds me of the other spectrum and the sizing, when you look at the electron microscope pictures and they always color them so that you can see different things. And then you realize, 'Oh, actually, somebody just painted that'. It's a little bit suspicious.

1:31 Emily Ruppe

I think it's kind of nice that we, as people, want to make stuff look pretty, that we color and paint the things that we see in telescopes and microscopes.

1:43 Miko Pawlikowski

That's how our brains work anyway, right? Every time you think about something, you recall that image and you slightly modify it. So I guess human perception, always adding stuff to what's actually objectively measurable. Speaking of which, of objectively measuring things. You work at Jeli.io and I'm pretty sure we'll completely and conspicuously wander in that direction at some point. But would you like to tell our listeners a little bit about yourself before we do?

2:15 Emily Ruppe

Sure. I'm a recovering incident commander. I was an incident commander for a while I've been doing incident command and response probably over the last like seven, eight years in my career. And I like most people who do incident response and analysis for their job ended up here completely by accident. And I started kind of in customer-facing support. I was doing weekend chat. I mean, I've been in tech for a lot longer than that, probably the last 15 years or so doing technical support, technical troubleshooting, things like that. And incidents were always like this kind of exciting thing for me, where it was very easy for me to kind of narrow my scope and really focus on what was going on and troubleshooting and communicating, asking questions. I really love asking questions. And I think when you're in an organization, and people see that you are drawn into incidents and like doing them, then they're kind of like, 'Ah, we've got one'. Let's have this person, you know, focus on incident response, samples with instant response, learn incident analysis, and things like that. So it was kind of my side gig for a very long time, in addition to the other stuff I was doing. I've been in hundreds of incidents. So it's not hyperbolic. I've been literally in hundreds of incidents, I've written hundreds of status posts, and probably hundreds of incident timelines as well. So I've done a lot of incident response and been in a lot of instant review meetings facilitated a lot of them and I kind of made my way into doing that as a full-time career. It was probably five or six years in when someone goes like, 'Yeah, and then maybe you can be like an incident commander, do this is your job'. And I was like, 'This is a whole job? This is a whole career?' And I had never really been aware of that. So I stumbled kind of into it as a career and yeah, came an incident commander and sort of working with Jeli.

4:12 Miko Pawlikowski

Wow. Yeah, that's the combination of keywords that you hear very often, you know, 'incident' and in the same sentence words like 'love' and 'excitement'. You did introduce yourself as a "recovering" incident manager, commander rather. Sorry, didn't mean to...

4:31 Emily Ruppe

That title it's, I have no emotional attachment to being a commander. It's okay.

4:35 Miko Pawlikowski

Yeah, sounds vaguely military, or navy rather. So yeah, I actually quite like that. Yes. So I can't say I necessarily share that approach to incidents. They don't necessarily invoke this, you know, feelings of excitement in me when I remember some of them. Can you tell me why you ended up liking them? I'm guessing everybody starts with, 'Okay, this is not particularly pleasant. It wasn't supposed to break. And now we have all this questions to answer to'. How did you go from that to, 'Oh, this is fun, I should do it more often'?

5:12 Emily Ruppe

I don't know. I mean, I've always been someone who thrives in chaos. But I think that there's something, it really does, like I can narrow in my focus. Because when an incident is happening, all of the other things that you have in the day to day of what you're trying to accomplish, and what you're trying to do, everything else falls away. And your main focus is, what do we need to solve this problem? What do the people need who are trying to solve this problem? What do they need? How can I help them? How can I communicate this information to folks who might be impacted by this issue? Like it just, I don't know, it feels like a very kind of, I understand that we have a problem to solve and our goal is to solve that problem. And I'm there to help facilitate that in any way I can. And it just kind of feels like a real, almost like I have blinders on it helps me focus. Because the most urgent possible thing is solving this. And I think that me starting to love it kind of came from the people that I was working with, and realizing that it didn't have to be this like horrible slog every time we were in an incident. We could joke and have fun and keep the mood light. And actually, we would be better at response, we would be able to respond to things quicker, and stay in facilitating conversation if we were having a good time with each other. And so that's kind of something that I started enjoying doing and bringing into incidents is just kind of helping folks take a deep breath and move through this. Because it's not incidents really are just high stakes troubleshooting. The things that every engineer has the skills to be able to understand there's a problem in my code, and I need to figure out what's going on so that I can fix it. That's a regular thing that we do. It just kind of reframing incidents as we're just troubleshooting. It just feels absolutely terrifying, because there's customer impact. And there's a lot of people at the company watching and seeing what we're doing. And kind of my role in that is helping make the rest of that fade away so that they can focus on the problem at hand and feel like they know what they're doing. Almost more like a camp counselor or cheerleader than commander was kind of always how I phrased it. That's why commander is not a title I'm emotionally attached to you. Because experts, the folks who write the code and run into problems, they know what they're doing. They just need help with the rest of it and getting through it.

7:35 Miko Pawlikowski

What you're describing almost sounds like and perhaps not almost perhaps entirely sounds like what athletes are experiencing when they are about to hit that bowl, or you know, start that run or do this race, right? Everything else kind of fades away and you focus on one thing. So I guess is that the state of flow really that's appealing to that? Or is it the satisfaction from actually knowing, 'Okay, this looks horrible, but I'm going to solve it, I've had 200 times before'?

8:04 Emily Ruppe

I think it's the flow, and there's also that kind of like you're going to get it, you don't really have an option to not solve this problem, you have to fix this issue. So that I mean, you might not be able to fix it in the most beautiful, complex, complete way, it could be that you are just slapping duct tape on so that the boat doesn't sink so that you can get to shore. But you're going to solve that problem. And when you solve that problem, you're gonna get that rush of dopamine. So you have like the adrenaline and the problem solving, you're working together, it's a team sport, and then you're going to get to a point where you have fixed something and are gonna get that rush too. I mean, I think there is like a literal like body chemistry, it's a little addicting. Because you're getting that rush and you're getting that kind of lift from solving problems.

8:48 Miko Pawlikowski

If anybody ever manages to make a graph of people's motivations, I think a lot of that will be dopamine in various forms, including incidents. Tell me, obviously, you know, from what you're describing, it sounds like incidents response is not a solved problem, right? We're not all enlightened to a point where we all do perfectly well on that. So what do you think is broken about how we do incidents in broad strokes as an industry?

9:19 Emily Ruppe

I think a little bit has to do with how we think about them. We think of incidents as this horrible thing that have happened to us, and that we have to react to that we have to solve the problem. Once the problem is solved, we have to make sure that it never happens again. And I think that's really an outdated way to look at incidents, because incidents are a good sign. Engineering in tech in whole, move fast and break things, right? That's kind of the phrase that a lot of our engineering organizations are focused on, move fast and break things. And when you break things, you learn how that thing came together and the way that we're kind of thinking about incidents as this thing breaks, we fix it, we make sure it never happens again. And then we never think about it again, is kind of backwards, because our systems are way too complex to be that, like straightforward with how problems occur, right? This one incident will have, you know, five or six different contributing factors, we might not be able to fix all of these different individual things in the grand scheme of things. So we put a fix in place at the sharp end so that it doesn't happen immediately in this specific way, this impact doesn't recur because of things that have gone on. But the entire organization, like your entire socio-technical system, the people that work with it, and the technical, they've all changed, the fabric of this entire reality is now different. So to think that we can now go back to where we were, when we started, we've solved it, and now we've returned to a zero place. That's kind of naive, right? Instead, we're operating in a new reality. And we've had this event that we've already paid the price for, you know, we bought a $400 juicer, we bought a peloton, we bought a very expensive treadmill. And we said, 'Ooh, this is hard. It's a lot of work for me. So I'm not going to spend a lot more time doing this'. When in reality, we're now leaving money on the table. Because there's a cost to incidence, in response, in what our customers are impacted and like actual money in the tools and things that we're paying for to actually fix the problems. To not sit and invest time in learning how do our people work together when they're trying to respond to a problem. How do our systems work together and in the ways that we didn't understand previously, but we were able to get to this issue. We're approaching these as kind of problems to solve, as opposed to like these rife opportunities to understand the whole fabric, the weave how everything connects in our systems to update our mental model of how everything comes together and works. So I think that we view incidents as problems to solve as opposed to these incredible, these just huge opportunities for us to delve into, discover insights about how things work and how they maybe could work differently or how they really actually work as opposed to the way that we say that they work or the way that we think they work. So I think that they're kind of this magic portal into how things are really transpiring. And I think that a lot of times, we miss out on digging into that, because we don't want to think about it anymore. It's this hard thing. And we've solved it. And so I don't want to think about it anymore.

12:31 Miko Pawlikowski

Right. So I think completely incidentally, that's basically the first thing that one sees when they go to Jeli.io. I don't mean the magic portal. I don't think they said that, but incident opportunities. Is that the biggest selling point? Is that really what you're trying to change? The perception of the incidents as learning opportunities versus you know, a terrible thing?

12:56 Emily Ruppe

Yeah, I think like when it comes down to it, I mean, it's obviously we are working on the whole thing, both response and analysis. But I think it's really just, it doesn't need to be as hard or as bad as I think a lot of folks perceive it to be. Actually when I was trying to decide whether I wanted to become a full time Incident Commander, I had a conversation with my brother, who I've actually worked with for a long time, too. And I was like, 'I don't know, this is kind of scary. This is like choosing a career path'. He was like, 'Well, what do you care about? What do you want to do?'. And I said, 'I don't want it to be so hard for everyone. Like when you go through an incident, I don't want everyone to have as bad of a time as they're having. I want it to be easier. I want it to be accessible'. Because it's not this difficult, scary thing. It really can be something that you enjoy and something that you get something out of. He was like, 'Well, gosh, seems like maybe you should do this as your job then'. But I think that's what drove me to want to come work at Jeli is I really do believe in the fact that like responding to incidents, and learning from incidents doesn't need to be this painful, difficult thing. And it doesn't need to be this kind of like horrible process that's being inflicted on you. It can be like a really interesting and enjoyable experience, especially when it comes to learning new things and discovering new stuff that you didn't know about the people that you work with, and the tech that you work with.

14:22 Miko Pawlikowski

That makes sense. I was wondering if you could share some interesting anecdotes. If you were to pick your most favorite incident ever, that was maybe the most entertaining or most unexpected. And obviously don't blame companies or anything like that. But which incident would you name your number one most favorite?

14:43 Emily Ruppe

That's really hard. This is actually a question I used to ask in interviews, is tell me about your favorite incident. I love the way that you phrase this. Because I think a lot of people immediately go mentally to like the worst incident they've ever been a part of, which isn't the question. It's what's your favorite one. And I have so many favorites. I have so many incidents that like, I am now emotionally bonded with those people. We have friendships for life out of those incidents. Though I would say my favorite one has to be, there was an incident where I was at a company, I wasn't even an incident commander yet at this point. But I was on call for incidents. And it was at the entire company kickoff. So multiple people from all over the country had been brought together to do our kickoff for the company. And we were on the 24th floor in a ballroom and a very fancy hotel having like a dinner. And it was it was very fancy. And it was a junior engineers' first time on call, they got paged, there was an incident, and we kind of swarmed at a table. And it was really cool, because it was people who wouldn't ever actually get to sit next to each other physically during an incident, because we worked in different states, that were all around a table working together on this incident. The only incident where we've had a professional photographer present, which I recommend, because we do have some really awesome pictures that look kind of like paintings of folks working together on this incident. But it was just really cool. It was a very cool experience to physically get to sit together with everyone, because I've worked with a lot of distributed companies, and walk around the table, swap beers out for water and have people that immediately swap water back for beer. Make sure kind of we were all eating. And it was it was really interesting, because I don't think everybody else noticed either that this was happening. There was an incident going on. I think at one point someone got up to speak, it was like, 'Hey, come on, guys pay attention'. And we're like, 'We're actively responding to an incident right now'. 'What? Is everything okay?'. 'Yeah, we got it, we're just gonna keep sitting at this table and working on this'. And it was really cool. It was a really pleasant experience. It was also kind of magical for that engineer who was on call for the first time, he was like, a junior Dev, first on call shift ever, and everyone was there to physically support them, and get them through the incident. It was just like a really neat opportunity to all be sitting together and served a very fancy meal while we fix the problem. I'm bringing like bags of flaming hot Cheetos to people, when there's an incident and said, we have some like fancy crew today and our d'oeuvres being delivered to us. It's pretty fancy, too.

17:22 Miko Pawlikowski

I love it. That's gonna be my benchmark going forward. When I ask other people, they'll be like, 'Oh, no, that wasn't that special. Come on, you didn't even have a ballroom'. Fair enough. And maybe another to extract from you, since I've got you here. So if you were to pick, maybe top one or two elements that you see probably the most that occur in most of those incidents, what would you say? It's probably the ones to always check out just in case. Any commonalities that are very visible from your experience?

17:55 Emily Ruppe

Yeah, there's actually some really interesting ones that I feel like we as an industry haven't yet really figured out how to navigate in the best possible way. Vendor incidents are always really interesting when a third party provider, because tech is now, we're built upon each other, right? We have so much of each other in the tech stack that we're using, and being able to collaborate with each other across company boundaries as incidents are happening is not something that we've quite fully figured out how to do. And I'm really interested to see how we start moving on that because it's a large reality in a lot of our lives where, 'Oh, this vendor is down. And my options are, do we pay an entire other vendor to provide the service for us now? Do we try and set that up and integrate it? Do we just kind of put our hands in the air and say, like, we are at the mercy of this vendors incidents, we're just gonna wait for them to come back online?'. It's a really interesting dynamic. I feel like a lot of organizations too, when it's a vendor incident, they don't spend a lot of time analyzing our response, or how we got to this position, because, you know, we can kind of very easily say like, 'Oh, well, this vendor went down. It's not our fault'. When in reality, there's always a lot to inspect about how are we integrated with this. Do we understand all of the different ways in different places that all of our different teams are integrated with this? Because it's very rarely straightforward. Everyone's using this thing in the same way. And kind of understanding what our options are around that. And the next is like our own internal tooling. Everybody's made their own stuff. And it's usually second class citizen, to be honest. Most of the internal tools that we make for ourselves, are not things that we upkeep. And it's usually really hard to get business prioritization to maintain and upgrade and make all of our internal tools that we use, operating in the way that we need to or even if we understand how they operate because they were made seven years ago by somebody who doesn't work here anymore. So it's kind of that inherited knowledge seems to be like a roadblock or a difficulty that we're running into. And it's oftentimes really hard to prioritize engineers spending time understanding things so that they know how they work, and they can work on them to fix them. So those are my top two.

20:12 Miko Pawlikowski

Absolutely. Thank you for that. So I think it's time for us to plug Jeli a little bit. So for everybody who's been thinking, 'Okay, well, obviously, this person knows what they're talking about. So I guess their company must be pretty good at that'. How did they get started? What do they go? What do they do?

20:29 Emily Ruppe

Yeah, jeli.io is our website. And our entire goal is to make incidents instant response and learning from incidents where accessible, something that you can actually do and isn't hard or difficult. And right now we have, our IR bot is available for free. I said right now, just in general. We offer incident response bot for free, we really feel like incident response bots are table stakes for every organization, just a way to get everybody in the same place and update folks that are not acknowledging the immediate response to what's going on. I mean, I've been on multiple teams that have built internally multiple incident response bots. So we're offering ours for free. And we also have a two-week free trial to use our incident analysis tool in Jeli. So our bot will get you through the incident, help you communicate stuff out, and then it will take everything that was in that slack channel for your answer response. And you can grab things from all sorts of other slack channels, where response kind of unfolded, pull that into Jeli, and we help you tell the story of your incidence by building a timeline, and kind of finding some themes and takeaways so that you can understand what happened a little bit more, and help your organization learn from this event that you've already invested time and money in.

21:44 Miko Pawlikowski

And that's jeli.io where everybody can find that. Emily, this has been really fun. But before I let you go, I wanted to ask you a question that actually asked a lot of our guests, because the responses are so fun and so varied. If you were to pick a single thing that you've done for yourself, for your career particularly, that provided you the highest return on investment. It could be anything, from a habit you picked up to a book your read, what would be the one thing that you would recommend other people try too?

22:16 Emily Ruppe

This is going to be such an embarrassing response. Have you seen the show "Ted Lasso"?

22:21 Miko Pawlikowski

I did.

22:22 Emily Ruppe

You have? Okay. There's a quote in "Ted Lasso", that's: "Be curious, not judgmental". And I use that on a constant basis. It has served me really well, especially for when I'm sitting being like, 'What are you doing? Why are you doing it like this? How could you think this was a good idea?'. I just kind of put on that 'be curious, not judgmental' cap, because I don't have all of their context, I don't know, you know, they are making the best decision they have available to them the information that they have. And I have found that approaching everything, not just incidents and incident analysis, but working with other people that I don't know very well, getting to know engineers. I've had a lot of situations where I'm meeting someone for the first time in the middle of an incident. And like that's been a really regular thing for me in my career. So approaching folks with that kind of curiosity and grace as opposed to judgment, because I don't know everything that's going on. But I want to, is really an incredible way to start building trust, which I find has been the most valuable thing in my entire career is getting to know people, trying to understand the situation they're currently working in and existing in, in order to find out how we can best work together and help each other has been for me, I mean, all of my career has come out of me being in the right place at the right time and being incredibly lucky because someone has said like, 'Oh, Emily can do that'. So that's what I found. Be curious, not judgmental, try and make friends and at the very least try and kind of understand what people are going through so that you can work best with them.

23:59 Miko Pawlikowski

Amazing response. And amazing show and look, people are already quoting that for life wisdom and hasn't even been that long. Emily, last question, for everybody who wants to follow your adventures, where do they find you? Twitter, GitHub?

24:16 Emily Ruppe

I'm on Twitter with immortal Emily, like immortal Emily, you can mortal enemy. The Immortal Emily. It's a play on my own name. And I will mostly tweet about like movies and then analyzing those movies as incidents, because that's a problem that I have.

24:40 Miko Pawlikowski

Fantastic. All right, Emily Ruppe everybody, Solutions Engineer at Jeli.io.