Hey everyone, my name is Abhijit, I'm a software engineer at Sentry, and welcome to my talk, Supercharged Production Stack Traces with Local Variables. So a little bit about myself, I'm Abhijit. I currently work at Sentry, and I help maintain Sentry's open source JavaScript SDKs. Sentry as a product helps you monitor your applications in production, take a look at the errors and performance data that's happening, so you can understand and fix problems faster. One thing we've learned at Sentry is that stack traces are really important to understand how and when errors happen to help debug them. You know this locally, and they're even more important in production if your server goes down or your site doesn't work. At Sentry, we try to make it as easy as possible to view your production stack traces. And so you can see that when an error happens on a site, or on a server somewhere, it gets thrown, Sentry, the Sentry SDK will capture it, and then surface it to you, real time, with the exact stack trace. So you can see the exact line of code that's causing your issue. But, the stack trace sometimes is an issue. enough in terms of what it gives you. There's still a vital piece of debugging context missing. That debugging context is what exactly the arguments or state was at the time the exception happened. not just looking at the kind of frames and what your code was doing, but what variables or classes were being instantiated at the time of your error. there's a solution for this, and a lot of debuggers give this to you. It's to allow you to see the local variables, or the variable state at the time of when a debugger is paused or when an error is thrown. So here is an example, I just attached a debugger to a really simple Node. js program, and it gives me the state of the variables that were defined at when I paused the debugger. We can actually grab this local variable state when an error is thrown, and attach it. Alongside the stack frame, so you can see this in production. And we support this in Sentry for languages like Python, PHP, and Ruby. But, for the longest time, we never really supported this in Node. js. Because there was no built in method to grab this data. There's no API for this, exposed by the Node or JavaScript standard libraries. And this is really important. a lot of people kept asking for this. It's very useful to debug an error in production. If you know exactly what the variables were defined as. But we always kept saying, hey, no, we can't do it, because it's just not part of the standard library. It's not possible within the JavaScript runtime. But, a little while ago last year, we learned that it is actually possible. We can leverage the same strategy debuggers use. and take advantage of the Node. js inspector standard library. And so the V8 inspector is built on top of the kind of, Chrome V8, debugger, and allows you to access a lot of information about the Node. js process, and manipulate it in ways to get this local variable information for you. So it turns out, via this inspector standard library, with a little bit of work and massaging the data, you We can use it in production to grab local variables and attach them to Node. js stack traces. let's walk through some code to see how this would work. here, at the start, we can, instantiate and start, the inspector and open it. this will start an inspector process, essentially a debugger process attached to, your main Node. js process. And In order to grab local variables from the inspector and attach it to stack traces that then we send to Sentry for processing, we need some kind of worker thread. So here we can spin up a worker, that really its job is to look at the inspector, grab the local variables and put it on, and give it to the main thread so that it can be attached to stack traces. so here is the debugger, that's being instantiated. in this debugger, we listen on a couple of events, which are exposed from the inspector standard library. most notably, whenever an exception is thrown, it varies if it's caught or uncaught, we pause the debugger and handle it. So essentially what we're doing is, When an error is thrown in your main Node. js process, we pause it, grab the application state, the local variables, attach it somewhere, and then resume it. And, although it can be weird, you're actually pausing the process, it's actually not that expensive to do this. here's what happens when we're actually pausing. So an error is thrown, the debugger, via the inspector, pauses the main process, We, and this is simplified. There's a lot of edge cases that you normally have to deal with. But what we're doing here is, going through making sure that, it's a valid error reason. And then we take a look at the frames of a stack trace in an error. Kind of grabbing all of the local variables for each frame and then attaching it to that associated frame. This is actually super powerful because it means we can access the local variables. Not just of the topmost frame in the error, but all of the frames wherever they're applicable. We then can use this inspector API to essentially call a fake function that will assign these frames and their associated local variables to a global variable that then the main process can then use later on when it's actually sending an error to Sentry. we have to make sure we release the object, otherwise, there would be memory allocation issues. what does getLocalVariables exactly do? It essentially uses, this runtime. getProperties method to be able to get the local variables via the inspector, we then have to massage the data a little bit. So to make sure that we deal with objects or arrays or similar, but really we're relying mostly on the inspector to do the heavy lifting for us here. So after we've written the local variables to an object and then the process resumes, the error is captured by the sentry SDK. Okay. and then transformed into an error event that can be sent to Sentry. what Sentry then does is it can look up on the error in this global here, and see, oh, were there local variables, and if they were, attach them to the outgoing error event accordingly. And, This all seems pretty complicated because you have to instantiate a worker, start the inspector, set up all these processes. But to a user who wants to start using this today in Sentry for Node. js, they simply just have to add include local variables true to their initialization and this all works for you. And here's what this would look like. So this is an example. Express app where I just threw a bunch of, I just threw an error with a bunch of local variables in scope, and you can see them directly attached to the event. Super powerful because now you know exactly what lines of code, plus what the variable state was when an error happened, so you can debug things even faster. while we were implementing this, there were a lot of challenges to work through. we had problems with worker overhead. Of course, we were instantiating a new worker process and we were instantiating this inspector instance. So we really had to make sure that the, memory and CPU levels were manageable. as we were, Writing things into the main process and pausing stuff. We also had to deal with memory pressure. unhandled exceptions was also something to deal with. There's different kind of behavior for pausing the debugger between handled versus unhandled exceptions. So calling new. error and explicitly handling that versus, throwing an error and letting it bubble up to the global error handlers. ESM versus CJS actually is still a problem that we're struggling with. They have, both have different behaviors and there's actually an active bug in ESM where, this doesn't work or local variable support doesn't work perfectly because of a Node. js bug. Currently, the local variables, setup does not support minified variables. So if you have some kind of build process, like transforming from TypeScript to JavaScript, and then you also minify, let's say because you're deploying to a Lambda or something. currently we don't support this, but the SourceMap spec is taking a look at how they can adopt Minified variables and minified local state and source mapping that. And so whenever that, is introduced into the source map spec, we can take advantage of that. If you are using this strategy, local variables, it obviously will, it's using a debugger inspector instance. So if you try to use your own to debug process, it doesn't work. So you have to use one or the other. And, unfortunately, because we're relying on this V8 inspector and the Node standard library, Currently, this only works for Node. js, so if you wanted similar support in the browser or for runtimes like Bunn or Deno, it doesn't work, but we're always open for ideas to see if we can make that work for those other. JavaScript runtimes and that's it. Thank you so much for listening in on how local variables really help with debugging, your production, JavaScript stack traces. Hopefully you learned a little, if you want to see the example code that was shown off and how this is written. There's a ton of edge cases, again, that we have to think about. this is all available in our open source JavaScript SDKs, linked at the very bottom. I have some links that you can reach out to me, if you have any questions. Otherwise, if you're interested in trying out Sentry and seeing how these local variables work, you can scan that QR code and use the code below to get started. Thank you.