Video size:

0:09 Miko Pawlikowski

Hello and welcome to another exciting episode of Conf42Cast, your neighboring galaxie's best podcast. My name is Miko Pawlikowski, I'll be your host today. With me today, Hammad Mushtaq, Software Engineer and Manager at Blameless. How are you doing, Hammad?

0:27 Hammad Mushtaq

I'm doing good, really excited about this first talk or first podcast that I've been on, tech-related. So, it'll be fun. Looking forward to it.

0:35 Miko Pawlikowski

First podcast, first talk. So, Hammad is giving a talk called "Pushing Code: Don't Forget to Flag Your Canaries?" at Conf42 Cloud Native this year, 2022, going live on April the 28th. You can subscribe below. And today, I'm going to try to get a little bit of info extra out of him. But before all of that, what's the most annoying thing about space travel?

1:01 Hammad Mushtaq

Yeah, I was thinking about this question. And, you know, what comes to mind is, it's just so annoying that everything takes such a long time to get results, when you're trying to explore. Like, I want to see results in my lifetime. And it's annoying that sometimes some of the initiatives just won't have results for like 10 years, 15 years, 20 years. And I think that's annoying.

1:22 Miko Pawlikowski

So, do you mean like the development time and the never-ending latest NASA rocket? Or do you mean the actual space travel itself and the annoying limits of the Physics?

1:34 Hammad Mushtaq

Annoying limits of Physics, right? Just figuring out, I think development is, you know, good. We have some technology today. But we're sending things out into space, and then getting a response back from them takes an X period of time, especially when they go really far in the Solar System, right? So, I think that's the most annoying part. You send things out, you build something today, sent it out to space, I don't know, it's gonna be like, depending on what I'm looking for, it could be years before I get a response back, an image or something that I'm trying to capture.

2:05 Miko Pawlikowski

That's definitely the case. Although we do live in interesting times with Elon sending satellites in batches and reusable rockets. And that is pretty exciting. So, things are moving forward. So hopefully, you have shorter response time soon enough. All right, Hammad. So, would you like to just introduce yourself a little bit and tell the audience a little bit about yourself?

2:31 Hammad Mushtaq

Absolutely. So like, because I'm the Engineering Manager today at Blameless, I've been with the company for about six to coming on seven months now. It's actually my first startup that I've been a part of. Before I've worked at fairly large corporations. My first job out of college was working for Microsoft as a Mobile Developer. So, I've come a long way where I'm not touching anything mobile-related whatsoever since I first started. I think one thing I would say about my career is that I jumped from an IC to the management track pretty early on, about three, three and a half, four years in to my C career. I jumped into management track just because the opportunity presented itself. And I made that transition. And I've loved it ever since. I'm really passionate about cloud and the opportunities that exist in that space today. Definitely started off in mobile. The peak era of mobile development, it was the 2013/2014 to 2016 time period. And then I sort of realized I kind of want to go beyond that. And so now I'm here working in some cloud and full stack development, learning such things now.

3:46 Miko Pawlikowski

That's very interesting. I always wonder if people who jump so quickly into the management track, do you not miss being hands on? How difficult was the transition for you?

3:57 Hammad Mushtaq

Yeah, honestly, it was, the first few months were definitely hard. I think there was four or five months where my mind didn't transition from being an IC and coming up with solutions, to transition to a manager, where I sort of guide my team to arrive at solutions. And that was the challenging part for me. I will say though, in my last company, where I became an Engineering Manager, I was actually really hands on for the first couple of years, even as a manager. So it was more, there was a lot more coding involved in that role, as well as the management aspect. And that was I didn't really fully move away from the coding, so I really enjoyed that. And now I'm in a place where I've sort of grown beyond that as far as team size and my roles and responsibilities. So now I'm not really coding in a day to day as far as my job is concerned. But that doesn't mean I'm not coding on the side for after work.

4:56 Miko Pawlikowski

So, you mentioned Blameless. I'm wondering how that differs. What made you want to jump ships for that? You know, like you said, it's first startup on your resume. How was it? What was the pulling, I guess, force for you?

5:12 Hammad Mushtaq

It resonated with me, the mission statement of the company resonated with me. Again, being an Engineering Manager, and you've been, you know, you've seen this, you've seen when you're at a company where you have incident response, and it's not really structured yet. And you're going to find that all companies go through this process where they're doing trial and error, and figuring out what the best way to do is their responses. At some point over time, what they arrive at is some process that's documented, right? As far as how to handle incidents, and I saw that over and over again in my last role, leading multiple projects, or multiple initiatives, seeing production issues constantly. I knew exactly the chaos that was involved in that. And, so when I came across, Blameless and I learned about the product and the solution and their vision, it clicked for me. In my mind, I was like, this is something that I would use, because as an engineering leader, no matter where I am, if I had this available, it would make it so much easier for my teams to handle incidents. And so it just made a lot, it just made sense in my mind to go to this company and help build the product.

6:17 Miko Pawlikowski

So, what's the mission? When you go to the website, the first thing you see is end-to-end SRE platform, what does that actually mean in practice?

6:25 Hammad Mushtaq

What it means is, essentially, we want to make the lives of SREs and engineering teams as easy as possible. That's the policy, just generalize in a couple of words. That's the terminology I would use. The on-call incident response is fairly stressful. When you come we talk about it, and you go through the process, it's really stressful for a lot of engineers, especially those on-call. And if you don't have guiding principles in place, it makes it even more stressful. So, the goal Blameless is to make that as easy as possible for those teams. We provide those rails and those guidelines. And you can, you know, obviously adjust your guidelines to your needs. But we provide the initial set of guidelines and tools and processes so that the engineers and SREs don't have to worry about where, when, how things are happening. They should worry about the actual incident and focusing on it. And that's what Blameless does. And end-to-end really just means we're looking at incidents, not just from when an incident starts, but before an incident happens from our SLOs. And the defining SLOs and SLIs directly in the product, triggering an incident during the incident response, again, all in the product, right? Syncing up Slack messages, etc. And then following up at the end, with a retrospective, creating extra Runbooks, etc. So, that's what we mean by end-to-end.

7:45 Miko Pawlikowski

So, I can understand the platform that, you know, hits close to home. But it sounds like a lot of what you describe is more cultural. So do you just deliver, you know, a big book of standards to the people and you expect them to learn? How do you, I guess, influence other companies cultural approach to SRE in general?

8:11 Hammad Mushtaq

I'm glad you brought up culture, because really, the word 'blameless' itself is the culture that we try to promote with the product. And I think the way we promote it is, again, stepping out of the way and making it as easy as possible for our customers to adapt the platform that gives you the step by step process in the platform, which sort of gets out of your way, facilitates, that automates as much as possible, so that the teams don't have to worry about it. Because when something is easier for a team to implement, they're more likely to adopt something like that. When something's hard for a team to implement, they're not going to adapt this and can be a lot of friction. So, our goal is just make that as easy as possible, as sort of invisible as possible. So, they can focus on solving the incident.

8:56 Miko Pawlikowski

You said 'SRE' at least five different times so far. Who's an SRE in the context of what you're talking about?

9:04 Hammad Mushtaq

SREs and engineering teams. SREs, when I specifically refer to it, are those individuals who are focused on maintaining the reliability and the availability and monitoring your services, etc. But really, everybody who has a stake, an engineer who has a stake in the reliability of your service, can be called an SRE at the end of the day, right? Somebody who has reliability and maintaining that service for their customers, making sure that customers have what they need when they need it without downtime, and then just have the experience that they're looking for in the product, that person can technically be considered an SRE.

9:37 Miko Pawlikowski

Fair enough. So, with that definition, I think a lot of people would fall under that umbrella. It sounds like in your talk, you're focusing on how to push the code and how to make sure that things don't break. Can you talk a little bit about what canary releases are and why they're important and what does it even have to do with Blameless?

10:01 Hammad Mushtaq

Absolutely. So, canary releases are releases where you're targeting a small set of users initially, with a certain feature. Your team is developing a feature, you want to roll it out. But you don't want to roll it out to everybody, you want to see how it scales, you want to see how reliable it is. You want to make sure you don't break anything or cause regressions. So you release it to a small subset of users, and let them play around with it, let them use the feature, you know, give you feedback. And then based on, you monitor your logs, you monitor all your metrics, and you observe whatever observability tools you have. And then once you've seen, okay, things are stable, things are looking good customers are using it, nothing's breaking, you know, got some positive feedback around it, that you will ask the next set of customers. And that's what you call a 'canary'. It's how you keep increasing the scope of customers over time, as you monitor and look at your metrics. We use this process internally at Blameless, so that's number one. We follow this exact same process when we release a feature, right? Now more so than we've done in the past historically, you know. We've obviously, as a startup, we've grown, we've learned from our mistakes, learn from where we've come, and we're following that more and more now. So you follow it internally. That's the number one thing. And second is that Blameless is there to promote reliability of your products, your services, and your teams in general, right? So, this supports reliability. When you start breaking down your features, and you start releasing features in incremental way and rolling them out slowly, you're helping improve the reliability of your systems, because you're making sure you're being very deliberate about when you release a feature, how you release, that you're monitoring your logs, etc. So, I think Blameless does, sort of, relate to that.

11:47 Miko Pawlikowski

Yeah, I always, when we talk about canary releases, I always think about the Canary Islands, but I think it's coming from the mines, right? When they would keep canaries as a kind of early sign of potential harmful gases. Okay, so I think what I'm really curious about is that, you know, working at a place like Blameless, where you effectively go and talk to different companies and try to help them with whatever they define as reliability, you probably have seen it all. So, I'm wondering, if you could give me a quick overview of what you see the most commonly, you know, being taken into account when people talk about reliability and how it's multifaceted?

12:34 Hammad Mushtaq

There's a lot of different factors. At the end, one of the things, one of the factors would be consistency, as far as process is concerned. And that's where Blameless really does fit in quite well, with providing a consistent way of handling incidents, consistent way of communicating to stakeholders that are involved. And I think Blameless facilitates that for a lot of companies. And that's really, really important for a lot of organizations that have a lot of teams operating, that they have a consistent way of handling things. So that's one, right. The another aspect of like reliability in general, is communication that I mentioned earlier. All of this ties together, being able to communicate more often, more efficiently, more succinctly, right? All that falls into reliability. If you can communicate not only to your internal stakeholders, but also to your customers in an efficient way, your customers see you as more reliable, because they know you're on top of it. Even if it takes a little bit longer. As long as you're communicating and maintaining that communication with your customer, you're going to be seen as a more reliable company. Because they'll trust you to communicate with them when there's a problem. And they'll trust you that you're going to do something about it. And again, another aspect of it is the small incremental releases or canary releases, again, demonstrating reliability at the end of the day, right? You're releasing something small, what you've done by releasing something small and to a small set of users over time is that a) you've shown that you have a process. You regimented a discipline about how your releases are being rolled out. And it just shows that you're constantly delivering to your customer, even if it's like a small part of a larger feature set that you've sort of, you know, say you have five features that encompass a epic, so to speak, right, and you release one feature toggle the rest off or feature toggles second one off, release one of them to the customer, then you release another one. This shows you're delivering consistently and reliably to your customers without breaking the experience that they're looking for.

14:38 Miko Pawlikowski

I think that the release cadence became a little bit of a vanity metric, because, you know, you see companies kind of showing off with, 'Oh, we've got 1000s of production releases daily', and at that stage it might become an overkill. But I'm wondering like from where you're looking at that you know, from the 10,000 feet view, where would you say is the bulk of people, or companies rather, in terms of the velocity of future releases on the, kind of, open market right now? Would you say that people have shifted towards smaller incremental, you know, like what you're describing with the canary releases? Or is the big, fat blogs still de facto standard from what you're seeing?

15:27 Hammad Mushtaq

I think it depends on which sector of the market you're looking at. If you look at a lot of enterprise companies that depend on some large applications, right? They don't want changes happening, because they rely, these large enterprises rely on these sorts of products. So they actually expect less cadence from their vendors. They don't want things breaking, like all the time. They want to know, okay, you're gonna release this feature, this time period, you want this release this feature at this time period, like in this scheme, q1, and q2, they want some systematic release cycles throughout the year, right? And then when you go to smaller companies that are starting up, like Blameless is a perfect example. We want to move faster, we want to offer our customers value sooner than later. And we want to show them that we are, because we're sharp, we have to show them that we are putting the resources in the engine and driving features out, driving fixes out to improve their experience, improving the tools that they have to do what they need to do. So we have to, sort of, prove ourselves and then doing incremental releases helps us and this helps us do that. If we just take six months, as a startup, where we're in a place from just competition, our customers will see us as almost doing nothing for six months, even if we promised them. You know, in the startup world, there's always promises you make to your customers say, 'Hey, we'll have this as our roadmap, we'll get this out this time period'. After six months, it's a long time, right? If you wait that long to release something, or even a month or two months is sometimes a long time for seeing, you know, a development coming from a startup that you've sort of trusted with your money and invested in for your company. So, I think it really just depends on where the company is, as far as their lifecycle. If we're early on, I think a lot of companies are doing small releases. Even bigger companies, if they have the feasibility and the trust of their customers to release more incremental releases, I think they do opt for and then moving towards more incremental releases, breaking up monoliths that they have built over time, and the breaking it up into service-oriented architectures, or microservices, whatever terminology you want to use, you know, whatever methodology you want to use. And they're opting for those more often releases, when they can get those fixes out to their customers faster as well.

17:43 Miko Pawlikowski

Okay, so you wouldn't venture an estimate of how the market is fragmented at this day?

17:50 Hammad Mushtaq

I understand. No, I don't think I know enough about all the different sectors in the markets yet to make an estimate right now. I'm sure there's numbers out there, the studies have been done, but I don't recall stuff.

18:02 Miko Pawlikowski

Another question. I've been wondering, when I was prepping for this, whether Blameless came up with the blameless culture or whether they were inspired by someone else. Would you happen to know who coined the term 'blameless'?

18:16 Hammad Mushtaq

I don't know exactly who coined the term. I do know that we base a lot of our concepts through experience, and also think, I'm assuming, I think LinkedIn was one of the first companies to officially have SRE teams found, like they were called 'SRE teams', and Google and LinkedIn, I think, one of the first ones to have those official teams on their own. So I'm assuming this concept of of blameless culture arose from some of the early adopters as they realized, you know, as they're finding incidents were happening, and they were going through a process of learning. That's my assumption. I'm actually not sure if we coined that originally or not.

18:55 Miko Pawlikowski

It would be handy if it was the case. Okay, so for everybody who at this points thinks, 'Okay, this Blameless thing sounds like a pretty nice tool'. How do they get started? Where do they go, what to do? Where is the free trial?

19:14 Hammad Mushtaq

To get started, you go to blameless.com. And you can submit a request to our sales team, and they'll get in touch with you and they can schedule a demo. Get you started and walk you through the product and see if something that your team would be interested in, learn about your team's processes, and tools that they're using. What metric tools are you using, and make sure that we can support those tools and then could you start, set up with a trial.

19:37 Miko Pawlikowski

Alright, so to shift gears a little bit. If you were to pick one skill or technology or language or whatever it is, that you would recommend looking into for our audience this year, in 2022. What gets you excited the most and don't say Blameless, we all know about that.

19:57 Hammad Mushtaq

Does it have to be related to SRE or can I say anything?

20:01 Miko Pawlikowski

No. Anything.

20:04 Hammad Mushtaq

Honestly, I think Web3 is something to pay attention to. I think that space is really growing. I'm not sure I agree with a lot of the, sort of the ideologies behind Web3 fully. But I do think that the technology itself is really powerful. And I think it's only going to get better. So, I think 2022, I think it's time to look at Web3. And there's a lot of like development tools for the developers out there to start, we're using Web3 technologies a lot, like more easily. You don't have to worry about setting up a lot of things. There's a lot of full stack development frameworks out there for Web3 that just make it super, super easy to get started and start playing around with it. So I think that's what I would say.

20:47 Miko Pawlikowski

When you say "Web3", what do you mean? You mean blockchain technology?

20:51 Hammad Mushtaq

Yes, yes. Web3 blockchain.

20:52 Miko Pawlikowski

Yeah, I guess I think if you asked 10 different people what Web3 actually was, you would get at least 10 different answers. And now it's getting even more complicated with metaverse. So, do we have metaverse? What do we mean about metaverse? What's going on? So, okay. All right. And the bonus question though, if you were to pick like one thing that you did for a career that was highest return on investment that other people could potentially replicate. And it could be anything from, you know, a skill you picked up, or a book you read, or a back scratcher that you bought on Amazon, what would it be?

21:32 Hammad Mushtaq

Definitely, I don't want to curse in this podcast. But there is a book that I really loved, called "The Subtle Art of Not Giving a F". And it's not related to technology. It's really just the general guidelines on how to live, like, live your life. And I read that, no, I actually listen to the audio version of it. And I took, I just found that super profound in the sense that it's something that everybody thinks about. But with the examples, this person who's very relatable gives in the book, right, very relatable, very normal, like, seems like an average person. He wrote a book about it, it just like it puts things into perspective. And I found that as a really good, it's just a really fun read and also just a good lesson to live like a good, sort of, a way to live by in life. And I think it's really powerful and contains, like the way you make decisions. And I found that really, everybody should read that.

22:30 Miko Pawlikowski

Fair enough. And on that note, everything good comes to an end. Today, my guest was Hammad Mushtaq, Software Engineering Manager at Blameless. For those who want to follow you, Twitter, LinkedIn? What's the best way to reach out?

22:49 Hammad Mushtaq

LinkedIn, probably. I don't have a huge presence on Twitter yet, but I probably should start getting one. But Twitter, LinkedIn, for sure.

22:58 Miko Pawlikowski

And for everybody who wants to go and check out Blameless, www.blameless.com. And like you said, you can get a nice free trial. Thank you so much for your time. This was really fun. And thanks for a book recommendation. I'll definitely check it out.

23:13 Hammad Mushtaq

Awesome. Thanks so much for having me. Appreciate it.

23:16 Miko Pawlikowski

And looking forward to your talk. It should be airing, once again, April the 28th at Conf42 Cloud Native 2022. Alright, thank you so much and see you next time.