Annihilating Your Errors

Video size:

0:09 Miko Pawlikowski

Hello and welcome to another exciting episode of Conf42Cast. My name is Miko Pawlikowski, I'll be your host. And today with me Finbar Fleming, Lead Customer Engineer at Rollbar. Hello Finbar. How are you doing?

0:23 Finbar Fleming

I'm doing very well Miko, looking forward to the podcast here today.

0:26 Miko Pawlikowski

Well, me too, that makes two of us. Thanks so much for coming. So a lot of ground to cover today with Rollbar and debugging. But before we do all of that, you know, our customery question for you to throw you off balance a little bit. So what is your most favorite thing about space travel Finbar?

0:43 Finbar Fleming

I think my most favorite thing about space travel right now is how much it's come down in cost. And that there are more and more journeys to space compared to, say, 15 or 20 years ago. Also love the fact that there's so many great videos and photographs coming back from particularly some of the NASA trips.

1:01 Miko Pawlikowski

Are you planning on going on a trip, when it becomes affordable?

1:05 Finbar Fleming

I would love to, yeah.

1:06 Miko Pawlikowski

We might get to that soon enough with everything that Elon is doing. I'm excited for that too. But you can't talk about space travel without talking about debugging, debugging rockets and debugging all kinds of other code. So, I guess my first question, when I was preparing for that, was what's, in your experience, the worst, worst thing about debugging?

1:26 Finbar Fleming

For me, the worst thing about it is that I get interrupted on what I want to do, which is working on a new feature, and I have to work on something or investigate something that is a problem that I've caused from some work I've done previously.

1:38 Miko Pawlikowski

That's very concise. Do you have any nightmare stories that you would like to share?

1:43 Finbar Fleming

I have many nightmare stories, particularly before using Rollbar. Definitely in the past, I've spent a lot of time working with old, you know, through log files, through system logs to try to find the root cause of an error. And definitely the other can be very time consuming. Definitely not made it any easier if it's in the middle of the night or on the weekend.

2:03 Miko Pawlikowski

Right. And, you know, you kind of name dropped already, "before Rollbar" sounds a little bit like, you know, BC.

2:11 Finbar Fleming

Yes.

2:12 Miko Pawlikowski

How does Rollbar change that?

2:13 Finbar Fleming

So, what Rollbar does is, Rollbar groups errors in real time, so it gives errors a fingerprint. So if you think of how you work with errors traditionally, maybe it's in a logging system, and an error occurs. And you have to go and analyze a log, maybe you heard about it from a customer. What Rollbar does is your errors get sent to Rollbar in real time, as they occur within your application. And the errors are sent with a lot of the information that you would typically have when you hit a breakpoint in a debugger, that error information is sent to Rollbar and the code flow, or that code path is given a fingerprint. And that happens in real time. And it opens up all kinds of different ways of responding to errors compared to traditional mechanisms that teams have in place today.

2:57 Miko Pawlikowski

A fingerprint. Okay, so how is that different from, let's say, just going through logs in Splunk. A lot of our listeners will have, you know, serious mileage under their belt with grepping through logs, how's the fingerprint different from that?

3:11 Finbar Fleming

So, if you think of working with an error, and an error has, it has a message and it has a particular codeflow. So there's something about that error that makes it different from other errors. And that's what Rollbar does, it gives that message and that codeflow, an identity or a fingerprint. Along with the fingerprint, we allow the developers to add data and to add their own context to the error. So, now you have structured JSON with a fingerprint, and that there essentially defines the error. And if the same error occurs again, it gets the same fingerprint. If a different error occurs, it'll get a fingerprint that has never been seen before. And you can then trigger an appropriate workflow based on the fact either that something is new, or a particular error that you know is problematic, is occurring frequently. And you have all this interesting information and context associated with the error that you can use to make a decision about how you want to respond to the error.

4:06 Miko Pawlikowski

So, it's really beyond the stack trace and the logs. It's all this additional context that lets you kind of go back and through. Does it also let you reproduce the situation later?

4:17 Finbar Fleming

So, we will capture as much information as we can. And yeah, a lot of our SDKs would support rerunning a curl command, for example, to reproduce the error.

4:27 Miko Pawlikowski

But, beyond the debugging, what other applications does it have? Is it primarily for just catching things that broke in the context of running that for your clients? Or does it also help other things like QA or the release velocity that I think I saw on your website at least five times?

4:47 Finbar Fleming

So yeah, definitely Rollbar can be used throughout the SDLC. So, the most immediate way that people tend to use us is that to address and respond to errors, either in QA or production. But moving on from that we would often tie in with automated processes that the teams have in place. So, if you think of maybe you have an automated QA process that runs during that continuous integration step, and at the end of that you could query a Rollbar to see if any new errors have been introduced. And you could, in an automated way, say that we're not going to allow this particular build to be promoted to production. And that definitely helps in terms of release velocity. So, when you're finding errors earlier, you're often finding the errors and being notified of the errors by the automated processes that you currently have in place. So, on average, what you're doing is you're finding errors more quickly, we're presenting the information back to developers in a way that they can quickly make a decision, quickly reproduce the error, and hopefully, then quickly get a fix that.

5:43 Miko Pawlikowski

And that sounds like a very good pitch. So, one of the things that I've learned the hard way is how difficult it can be sometimes to get developers to change their ways, and to get this initial buying for a new tool. So I'm wondering, how do you kind of describe the unique value proposition for choosing Rollbar over a variety of other solutions that are available in the market?

6:06 Finbar Fleming

So definitely, the unique value proposition is that the speed and the accuracy at which we group errors, and what that means for a developer is that they get notified of errors more quickly. And that the information is presented to them in a way that they can make a decision. And getting back to the contextual information that's in the air that you can actually for many categories of errors. And in fact, chaos engineering is pretty much based on this, is that there are many categories of errors, that there's something about the error, that means that you can automatically respond. Maybe you're automatically turning off a feature flag, if there's an error associated with a feature flag. Maybe if there are errors associated with writing to a message queue, you have some process to restart something in the background, so that that frees up. It may be that if errors are occurring immediately after a deployment, that you want to automatically roll back. So the unique value proposition of robar is facilitating all these workflows and these automations that either teams may have in place today, or at the very least, it's a goal for them over the next 6, 12, 18 months.

7:11 Miko Pawlikowski

And then from what I remember the actual company being started, it's also in a fairly unique way, right? Would you mind talking a bit about that story?

7:21 Finbar Fleming

Yep. So, we were founded by two developers. And they realized that when they were interrupted, when they were working on errors, that it was completely disruptive to their efficiency. And essentially, Rollbar is a platform to address that issue. When errors are sent to Rollbar, they can be fingerprinted., they can be identified, you can set up a process for this. We know that this error here has occurred before and we know what the appropriate responses for this error are, or we know that a particular error is new, and maybe needs to be looked at. And that there's data and interesting information associated with an error that can be easily captured at the point the error occurs. That allows you to understand, do we need to fix this right now? Or can we let it wait until tomorrow or let it wait until the developers finish the task that they're working on, so that you're not interrupting somebody mid-task, and instead, that they can then just pick it up as a medium severity issue, when they finish whatever work that they're working on.

8:16 Miko Pawlikowski

Speaking of technical stuff, two developers starting a company and a product for other developers. Can you tell me a little bit more how it actually works behind the scenes? Because my naive understanding, from looking at your website, is that in order for it to work with different languages and frameworks, it has to have a certain understanding of producing quality fingerprints, you said.

8:38 Finbar Fleming

Yeah.

8:39 Miko Pawlikowski

Does it have like a separate support for every language that you support at Rollbar?

8:45 Finbar Fleming

We do, yeah. Behind the scenes, what's happening when an error occurs within within your application, a JSON representation of that error is created in a particular format. And that data is sent via an API call to Rollbar. Now, for the languages that we support, we do optimize the the analysis of those errors, but it will work for any language, even languages that you don't see on our website. So, for languages that we don't have an SDK for, you could in theory build your own SDK and send the error information to Rollbar. But for the languages that we support, and there are many, we pretty much cover all the main development languages, we do have optimizations in place for each of those languages. What's happening behind the scenes is that, for a percentage of the errors that we analyze, we run them through a secondary process. And it's a machine learning algorithm that identifies potential new clusters of errors. They then get reviewed by our R&D team. They get run again in parallel for a period of time in QA, and then rolled into the product if they're seen to improve the overall quality of the grouping.

9:47 Miko Pawlikowski

I'm guessing using Rollbar to debug your Rollbar?

9:51 Finbar Fleming

Absolutely. Yes. Rollbar. Yes, we use Rollbar to analyze the errors in our own platform. Absolutely.

9:56 Miko Pawlikowski

I also noticed that you mentioned in your website that you're using AI. And I'm typically fairly skeptical of people saying, 'We're using UI for everything'. So I'm curious of how much of, I guess, improving the quality of the fingerprints can be done automatically through basically artificial intelligence? Would you mind talking a little bit about that?

10:20 Finbar Fleming

Yeah. So, to reiterate, yeah, we run a percentage of the errors through a secondary process that identifies clusters of potential rules. Now, we do run them through a manual review. So R&D do review them before they're confirmed and accepted. And then they're tested for a period of time before they're added to the overall product. And then on the note, the response side, we would have AI in the area of deciding when to notify a team. So, one of the problems that development teams often need to work around is that there are many errors already in their applications. And there's sort of errors that are expected, the worst kind, yes. And really, you don't want to interrupt people unless it's necessary, and you're looking for some anomaly in the pattern. And we would have AI in that area as well around helping people to decide when they would be notified of errors.

11:15 Miko Pawlikowski

Right. And are you also in the business of actually notifying people like pager duty style? Or is that left to other integrations?

11:23 Finbar Fleming

So yeah, so we have many integrations. Yeah, there's sort of a few flavors. One are things like chat systems like Slack or email or HipChat. The other would be notifying bug tracking systems. So, under certain circumstances, you may want a defect automatically created or an issue automatically created in JIRA. And then tying in with APM solutions like Datadog and PagerDuty, so that you're notifying those systems when errors occur, or particular types of errors or a particular type of error occurring at a high frequency. And then they're making the decision around who to notify. The last area is in a web hook integration. So you know, if you had some custom response that you wanted, you can easily add your own custom response, based on a particular pattern of errors, notifies your webhook under certain circumstances, and then you trigger your own automated response.

12:13 Miko Pawlikowski

And all of that, I'm guessing from the point of view of the SDK, they work as libraries, right? They're built into the application itself, rather than like a Daemon running somewhere?

12:24 Finbar Fleming

Absolutely. Yeah. That's a big differentiator actually, is because we're running inside the application, we have access to information within the running application. So yeah, so each of our SDKs essentially is a wrapper around our API. And it'll be slightly different based on the individual language, based on the requirements of the language or the capabilities of the language. And we capture the information processes, and then send it off to Rollbar for analysis. But yeah, it's imported, like you would import any other library, just add it to, you know, a Maven, or Gradle built file, package.json file. Just add the library and then instantiate Rollbar startup, and then you're up and running, and you're sending errors to Rollbar. The initial configuration should take no more than 15 or 30 minutes. And then as you add, and we do recommend that you build out some of these integrations to the important tools in your SDLC and in your continuous integration or on deployment processes to allow Rollbar to help you to make decisions throughout the SDLC.

13:27 Miko Pawlikowski

That makes a lot of sense. Let's move on to a story. Everybody loves a good anecdote, and I'm definitely not letting you off the hook before give me one. I noticed that you have Duolingo and Bubble, both applications on which I spend more time that I would be comfortable admitting online.

13:45 Finbar Fleming

Okay. Yeah, no, it's great. Yeah, they're good. They're both interesting customers, and that they're both customers who are deploying large amounts of software, big engineering teams deploying frequently and Rollbar is part of their process.

13:59 Miko Pawlikowski

Yep. That's what they say on their website. So, can you give me some of the more interesting stories? Things that people might have discovered using Rollbar, some more interesting bugs that were prevented from being shipped?

14:11 Finbar Fleming

Sure, yeah. So yeah, so definitely we have a number of customer stories on our website that people can check out. Interesting things like, you know, just the amount of money that people save and the reductions in the amount of time teams spend debugging. Interesting story of, somebody started using Rollbar, very quickly realized that there was an error in the shopping cart area of their product that impacted a particular browser version that had been going on for essentially a long time before they started using Rollbar. And literally as soon as they turned Rollbar run in production, they found this error and obviously that made a big impact to their revenue.

14:50 Miko Pawlikowski

Okay, so for everybody who's just waiting now impatiently to get started with Rollbar, they go to rollbar.com And what do they do?

14:59 Finbar Fleming

So, you go to rollbar.com, sign up for a free account, and you get 25,000 events per month forever.

15:05 Miko Pawlikowski

Wow, I like the 'forever' and I like the 'free' part. And for everybody who likes forever and free and also likes food, I was told there will be a lunch and learn.

15:15 Finbar Fleming

That's correct, yeah. People can sign up for a raffle, where their team can get a lunch and learn where they learn a little bit about Rollbar. So, definitely sign up. If you think Rollbar would be useful for your team, sign up and you'll be in for a raffle for lunch and learn for both you and the rest of your teammates.

15:34 Miko Pawlikowski

Fantastic. I will put the link in the description below. And on that, I think that we can almost let you off the hook. But before that, I want to squeeze a few more more so of wisdom, I guess. If you were to recommend following a single technology or a language or some tech thing, that's not Rollbar, in 2022, what would it be?

15:59 Finbar Fleming

Good question. I think I would be loath to tell anybody to focus on any one technology or language. Definitely some of the things we're definitely seeing traction with Flutter in 2022. I would focus on just making sure you have a good basic understanding of post offers developed and then roll with the changes.

16:18 Miko Pawlikowski

That sounds like sound advice. And the last question I had for you today, if you were to pick from your tech career a single thing that provided the highest return on investment that other people can replicate and benefit from years of experience, what would you recommend, or what worked for you, probably?

16:40 Finbar Fleming

I suppose the big thing that in my career, the big thing that made the biggest change was understanding that you can have different styles of development to deliver the same product. And very early on in my career, I was introduced to agile development and the fact that you can deliver software in a way where every time you make a change, an automated test gets executed. You can maybe program using extreme programming techniques, so that you don't get these silos of information. And compared to the way that I had been developing software and the team that I had worked with the developed software, that was a huge, huge change. And I would tell people to understand the process, understand that there's no one way to do things and there are always ways to improve the process of which you essentially delivered the same end product.

17:34 Miko Pawlikowski

And that's sound advice again. Thank you so much. Everybody, Finbar Fleming, the Lead Customer Engineer at Rollbar, that you can go and check out for the free forever 25,000 events per month at rollbar.com. Thank you so much, guys. It's been a pleasure. I'll see you next time.

17:50 Finbar Fleming

Sounds good. Thank you.