Conf42 Chaos Engineering 2021 - Online

Día de los Muertos - Postmortems that saves lives

Video size:

Abstract

I’ve been developing software for the past 20 years and managing teams for over 15 of those. My current team has grown from 5 to 18 over the past two years whist our application grew from a few thousand to more than a million users. We had our fair share of issues and we always kept our heads in check on how to improve things. Our way of doing postmortems is something I’d like to share with others. Not only because I believe it’s helpful, but also, because I love to receive feedback from outside our team.

Postmortems are a well-established way to document the history of a project, especially when things break or don’t go as planned. Most teams have a hard time keeping up with it. Among those who do, to get value out of it is also another challenge. Let me share how my team brought postmortems as part of our process. By not only making it a must-have when handling emergencies but also celebrating it we transformed it as a tool for team bonding. Also, bringing postmortems as an onboarding tool to newcomers. To finish, I’ll share two occasions where our postmortems helped us avoid issues and paid themselves.

Summary

  • Fabricio: I think we can do more with postmortems in the tech community than we are using today. He says we are always focused on the failure part, but there is more for us to extract from it. Fabricio is the CTO at BXBlue.
  • A post mortem is a process used to identify the causes of a project failure. It's a tool for recording the history of a company, of a product, the tasks and the moments that were shared. Here are some techniques that you can use to improve on your postmortems.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
You. Hi. I'm going to talk today about postmortems and why I think we can do more with this technique in the tech community than we are using today. First, let me introduce myself. My name is Fabricio and I've been coding since 2002. I've worked with big corporations and also small startups, and I had my time in academia as well. Nowadays I'm the CTO at BXBlue and this is the experience that led me to what I'm trying to show you when I things about postmortems, I see in the community a grim face about it. Even though there is this shared knowledge that it's a powerful learning tool, we are always focused on the failure part, but I believe there is more for us to extract from it when we look at what we can celebrate and what we can improve on it. So let me share about it first. Looking what is a postmortems? According to the dictionary, a postmortems is an examination of that body to determine the cause of death. This definition comes from medicine, but it's pretty close to what we think about postmortems when you're looking in tech as well. It's not a surprise when you look for the definition of postmortems, according to the PMBok, is that it's a process used to identify the causes of a project failure and how to prevent it. So it's similar to what medicine defines, with a twist that we're looking here not only to determine the cause of death, but also to learn from it and avoid it in the future. The counterpart of a postmortems is a pre mortem, where instead of looking to the past, we're looking to the future. And it's a process used to foresee the possible causes of a project failure and mitigate it. So it's a risk analysis that we're trying to improve on what we are doing and avoid those risks in the future. But when you want to look forward and you want to try to foresee those risks, you have to learn from it in the past. So a post mortem is one of the biggest tools to do a good premortin. And when you're looking for a post mortem, for me, the best definition of a postmortems comes from its synonym, autopsy, which comes from the greek sing for yourself. So it comes from experience, yourself, your mistakes. And for me, this is a good postmortems. It's a tool for learning. It's a tool for recording the history of a company, of a product, the tasks and the moments that were shared. So what some people experience it, other people can also experience as well and share this context, share these learnings. It's a communication tool for all the teammates in a company. But the types of postmortems that we are used, CTO, it's very limited. So I'm going to share some other types, some other techniques that you can use to improve on your postmortems. So I classified six different types of postmortems, including the classic ones and some new ones. So the first one is the classic one. Those closure is the one defined for the pinbot. It's not restricted, but usually started for project and it's focused on reporting what succeeded. So you can double down on that and what failed, so you can learn from that and avoid that in the future. The second one is the root cause. Analysis is the most common one. When you see a tech blog about a post mortem, usually it's targeted on failures and downtimes, and it's focused on reporting why it happened and why it won't happen again. It tries to give the sense of security that you know how to fix it or that you already fix it and it won't happen again. A new one that I want to introduce is the detective. This postmortems tries to recreate, usually in a first person, a step by step description of what happened. It's a good way to share with other people what were your process of thinking when you're handling with a situation. This will allow other people to look on your process and learn from it, or even allow you to improve on your process itself. A different type, and is a very uncommon one, is the open problem. It reports those findings of an ongoing issue or a pain point without an implemented solution yet. So here we are talking about something that is not fixed or even you don't know the solution on how to fix it. But it's very useful because you're onboarding what you learned and inviting other people for either contributing with the design of the solution or the investigation of the problem. Or even if it doesn't come CTo be solved right now when it comes back in the future, you know where to start from and you don't start from scratch. And those last one, and not least, is the celebration. I'm pretty sure that you celebrate your achievements with your team, but usually these are shared celebrations and they are pretty much already defined. But each person has a different learning curve and each person can have a different point of view. But what is an achievement for them. So celebrating about why things are important, it's very helpful on recognizing that each person has a different perspective on the team. So with those different types of postmortems, what can you do? CTO keep up and do it in your team. So first, schedule some time for it. Be it after a big event or something that you think that you should record that learning process or that context. Schedule some time and just write it down. Then agree with other people where you should start it. So other people know where to look for context and where to find what happened and when. Then after you write it down, collect some feedback, not only to share what you learned with other people, but also to see what they think about what you're writing. Also, they can share some light about other things that you didn't notice and you also could learn from your experience. Then share it, be it right away or afterwards. When you see something similar happening. Share your postmortems so other people can also learn from it and get some value from it. And also the most important, just start doing it when you start doing it. If you don't have the other parts already figured out, they started fitting CTO place. So this is it. I hope I shed some light about postmortems for you, and I hope your interest had a little push forward about starting yourself trying these new types of those postmortems. I'm very eager to learn about how you're doing your postmortems and how they are helpful or not helpful or your challenges, and I'm very open to discuss. So till next time.
...

Fabricio Buzeto

CTO @ bxblue

Fabricio Buzeto's LinkedIn account Fabricio Buzeto's twitter account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)