Hey everyone, I'm excited to be talking today about a subject that's very close to my heart and, and something I've been thinking about for a while and have been working on. And this is diving into this question of why browser devtools don't understand your code. And just to give some background, I've had a lot of experience and I think many people in the audience are familiar with these development environments like Visual C Plus plus code and then these game engines that understand your code, meaning that while you're devming you have some kind of visual representation or something in the devtool that really works with your code in the terms of your code. And conversely, we see when we're working in the web using JavaScript and typescript to build front ends, we have tools that don't really understand our code. They do other things. And in this talk I'm going to be talking about why I think this has been historically the case. And I'm also going to be talking about this project that I've been working on recently called Mighty Meld, which aims to close that gap. So let's get a little bit into the history. My personal experience when I started developing early in my career, I was doing Windows apps and I got used to this Windows form builder where you could build the UI visually in the devtool and then connect that really elegantly to code in Xcode. Building iOS apps, there's this thing called interface builder that I used quite a bit, which allows you to visually structure your app using a visual tool and then sort of code at the same time using a real programming language. The experience that I think is actually the best in this regard is what I've experienced with the Unity game engine. And this is pretty standard practice for game engines. And I'll just hop right now to this video so we can just get some visuals of what I kind of think of is like a kind of peak experience. So basically this video is from game makers toolkit channel, and they're going to be going in and basically making a game over screen that appears on their game. And the way you do this in unity is very visual and very intuitive and also very powerful because you can code anything you want in this system. And as you know, games do a lot of things that require quite a lot of very real code. And so you basically see here they're adding a game over text and now they're going to be adding a button and you just kind of drag these things on, use visual tools to set them up at no point are you messing with CSS, going back and forth, hopping through like many text files to size something, you size it directly, which I think is a great, I found to be a great experience working here in this environment. And you can also, if you want to do an on click handler, you can see here there is no on click handler. And now they're basically going to go ahead and hop over to the code. And you see here, everything's in c sharp. And you can go and add your event handler, in this case restart game, go ahead and import some deep unity guts on the scene management as a package. I'm not sure what it's called in C sharp actually. It's been a while, but anyway, call this load scene function basically when the callback is triggered, and now you can go and go to your on click handler, link it up to that actual piece of code, and then without all wired up, you can basically play your game and see it all happening right there on the screen. And this is what I think of as a peak development experience. This was my peak development experience when I worked for Unity. I didn't work for Unity, sorry, when I was working on a lot of unity games and some VR projects in unity as well. And it's really cool actually. You can even pause the game while you're running it and look at all the stuff contributing to what you see on screen and understand it visually in a way that's code aware. And many years ago I transitioned from Unity to doing web work. And ever since then I've been asking myself, why can't the web have something like this, this way of working? Because the way we work currently in the web is you get typical view. You have your browser, and of course you have your vs code or your text editor. But while you're looking at your app, you have this view, which is the inspect view, the typical chrome devtools inspect view, firefox. All browsers have something like this, where you can see the Dom of your app as you're working on it. And this is certainly useful, but it's not very close to your code. And as apps have gotten more and more and more sophisticated, the gap between the Dom and what's in your code is getting bigger and bigger. In the react world, we have the react devtools, which show you can actually show you some of your components, but the react devtools are also something processed. This is a bunch of stuff here that I don't see in my code at all. And so the picture that this devtool shows me isn't really a code accurate picture. And this has really been something I've been thinking about for a long time and started working on this project because I just feel like we can do better in the web. And then you start to have to ask yourself the central question of this talk. Why don't we have this yet? Why don't browser devtools understand your code? And so before I start getting into the way, I think we could fix this, let's talk a little bit about the history and how this came to be. If you're an old timer like me, I think I qualify as one, because I actually remember Dreamweaver when it came out. And a lot of old timers, when I talk about this, they say, dreamweaver that failed, can't be done. And Dreamweaver actually didn't fail. It was a really successful product in terms of as maybe the first no code tool, but it didn't achieve its aim of making an environment where you could go back and forth between a Photoshop like visual experience and real code editing. And that was what Dreamweaver was trying to aspire to. What ended up happening in practice is if you started to make a complex site, you started layering some javascript and other stuff like that, and working a lot. In Dreamweaver, the code got progressively more messy the more you worked in Dreamweaver, and so it wasn't really suitable for this back and forth between code and design. And also if you wrote a bunch of code, Dreamweaver wouldn't necessarily understand it, and things just kind of fell apart. And so this dream of having a real synchronicity between a visual tool and code goes back. This is about 20 years that remeaver goes back. And a lot of old timers are like, we tried it, it didn't work and can't be done. Is the corridor follow one of those? If you start thinking in that way, of course we can think about things afresh today. But anyway, I dont want to paint Dreamweaver as too much of a failure, because Dreamweaver was actually a massive success. They just sunset the project a couple of years ago. I think it started in the late ninety s, and Dreamweaver was really the forerunner of a lot of these no code tools and things like Weebly Wix, squarespace and webflow. These tools where you can work visually in a visual tool and generate code. And these have been a tremendous success and are really great for people who don't know how to code, want to make something that simple. But of course, as all we know, we're, I think, pretty much everyone in the audience here, Javascript developer or in that ecosystem, there are a lot of things that you need code for. You need certain excellent performance, you need a little lot of sophistication, you want to bring together a lot of the third party libraries, you want to connect to your certain microservices or your CMS layer. Once you enter the world of code, you can do anything. And when you're building really complex and sophisticated projects, or maybe smaller projects that need certain things to happen very well, we turn to code over and over again because these platforms don't give us the flexibility. So no code is great. But obviously as developers we want something that works with our code. And these sort of design to code solutions haven't been able to go in the other direction. Another answer that an old timer will tell you, not quite as old as remover, but shortly thereafter there was this decade where Flash was driving a lot of the experiences on the web, and Flash has a flash actually really succeeded at unifying code and the visuals. And a lot of people who were building things during that time remember very fondly how amazing it was to be able to work with an artist and have this sort of visual tool that could help you develop that also was tightly connected to your code. Flash didn't make it through the transition to mobile for performance reasons. Another thing that's interesting about Flash that I think is especially pertinent to sort of where I see some of the challenges of making this kind of devtool is Flash's shortcut. So Flash did everything in the early days, you would have a flash movie, and so it's kind of like canvas today, where you'd have a box where you would be drawing your flash content, and that's where everything lived. And if you're working in the authoring environment, you'd be authoring something to live in that box. And that actually makes things much, much simpler. Flash tried to make the transition to being sort of anywhere on your web page more into this document model we see in the web, and they didn't really succeed with that. There's this thing called Flex that I use some and was a little bit successful, but it didn't seem to really have that same kind of dynamism of just the pure open web with everything, open standards, all kinds of stuff popping into the page, being dynamically loaded with different technologies. And so I think Flash didn't adequately solve the problem in sort of what we need for the web. So I think that's kind of why Flash isn't around and why something like Flash hasn't taken over to provide this for us. But the third answer to the question and the heart of my talk is, actually, we can do this. Now, we might have not had these in the past, but the time is right for web devtools to start to understand your code and allow you to work visually with it. And core to making this work is to have some kind of part of your code, some visual part of your code for the devtool to work on. And so this is again, kind of like why this is easy in other environments and difficult on the web in other environments, whether it's unity interface builder or flash, you have this scene file, which is basically this thing that has all the geometry and all the layout in it, and it's closely tied to the UI. And the other thing that these environments have going for them is that they're closed, they're owned by someone like unity, Apple, Adobe, they own these various environments. And so if you have this closed environment and a scene file where everything is in these xy coordinates, it can become pretty easy to then make it. Well, not just easy, it's natural. It happens over and over and over again to make a dev tools that acts on that scene file. The challenges with the web are twofold. One, what is the scene file for the web? What dictates what things look like separate from the functionality on the web? And then the other thing is like, well, the web is no one controls other standards bodies, standards bodies and corporate groups and things. But the web is kind of an open, anarchistic environment where there's no one dictating how things get dont. And so sort of back to, before we get into my answer as to why I think this can be done now, in the whole history of the web, it's been a constant challenge. And if we look at the evolution of what you could call the scene file for the web, actually, in the early days of the web, in Tim Berner's Lee time, when he was originating the web, there was kind of a scene file. HTML is a basic visual description, and you could see devtools working really well on that. Actually, Dreamweaver, if you only have HTML, is a really pretty good tool because it's pretty one to one with what you're working on under the hood. The challenge with the web is that very quickly people started adding CSS and JavaScript to HTML in order to have functionality, global control of styles, all the nice things we kind of take for granted now in like the mid to late ninety s. And that made things a lot more challenging to separate the visual layer from the functional layer, to sort of break out a part of the code that you could kind of see visually and that would relate to what you saw on the screen while you were running your site and in the industry. This is I guess probably mid to late 90s that these things came in. And sometime around the early to mid aughts, NBC started to become really a dominant way of thinking about the web. So thinking about when Ruby on rails appeared and things like that. Cake, PHP and various NBC frameworks started to have these template layers that abstract away some of the visual. And then there were actually like template builders you could get, but it didn't really get you tools deep into the app because the templates were very, very shallow visually and not as connected to what was happening in the rest of the app. And then about ten years ago, with the advent of react, we got this thing called JSX. And so JSX is actually very powerful and very clean way of separating what appears on the screen from what doesn't appear on the screen, in a way that's very connected to functionality and also the sort of component, the way components work of really bundling together functionality with display in a way that's building block like. And so my sort of thinking as to why it's taken so long for us to really be able to do this on the web has a lot to do with the time it took for JSX to kind of appear and get to a point of maturity. And now that we're here, this kind of thing seems very possible, and doesn't just seem possible, I've been working with some people, we've been building this thing called Mighty Meld, which is a code aware devtool that basically understands your code. And by your code I mean the JSX and the style part of your code. So I'm going to talk a little bit about how this works and how it all can be done in the sort of open standards world of the web. But before I talk about how it works, I'm going to give a brief demo so you can see kind of what it is and get a feel for Mightymeld. So let's hop over. I have here a react project. So it's basically like standard react project with components. It's built using shocker UI. And what I've done to this project is I've added the Mightymeld runtime. I've set up plugin in veat and then I've set up this config file. Actually the installer set this up for me, but it basically has a command for how to run my dev server. And then with these things in place, I can do NPX mighty meld and it'll pop up in this mighty meld studio. And here I see my app running. And so here I can see my app. And actually I got two versions. Let's close this down for a second. So my app is actually running here in Mightymeld and this is just an iframed version of my app. And so far this is actually pretty normal way of developing. You can even use your traditional chrome devtools if you want to understand what's going on in the Dom or to see the console. But if I click this edit button, that's where things start to get interesting. As I click around, I see not the Dom, but I see actually my code. And so you can see there's this react tree over here. I'm clicking in the sidebar component. As I click on different things, the pieces that drive them highlight, everything is component aware and also sort of JSX aware. So this is actually being drawn in a map. And I can at any point hop over open in editor and basically see what's going on here. So you see this thing on the left is a representation of my JSX and includes things like Navlink's mapping. Every item is this map. And so when I select this, it's one thing in the JSX and four things on the screen. As you would expect from the way it's written, I can also see props that are put on this same props you see here appear here. So you can see the props. And for standard props you get nice drop downs where you can work with things. And so the first thing that this development environment gives you is a way of inspecting your app that's code aware. So you can inspect, you can see what's going on and everything's phrased in terms of your code. Super useful for hopping into a new project and trying to understand what component drives what. And as you can see, as I click around, this is a tv series component. It has a movie list. I can either drill down this way or I can drill down that way to drill down deeply into the movie cards or even like the bookmark icon and see how everything is put together right here. And so it's pretty rich inspection experience. From there I can go and change things around. So to the extent that things appear as JSX or styles, they can be edited. So I can call this my movies and it updates the app right there. I can duplicate this or be there twice or delete it. And basically as I'm doing this, you can then see that these changes are happening to the source code on my local disk. So it's a devtool flow. It's just kind of doing the same code updates I would do if I was typing. And then you can move things around. Let's say I wanted to take this and put it above like the title above this information so I could drag it up here and it reorders it. And then as I'm building, I can hop over into drive mode and experience the app and see how things feel. And then maybe I notice, oh hey, these are inconsistent. So I can hop back over to edit mode and move this one up as well to make it feel consistent because I guess this is a different component, right? This is the movie trend card and this one is the movie card. And so basically using this, I can go in and make changes. And I can also go in and make changes to styles. I dont know what kind of style things can we do here? It seems like. I bet we have, yeah, we have a gap. Like for example, maybe I want to reduce the gap between these so I can go ahead and make those, those style changes. And then we also have some building blocks set up here along with the building. What? These building blocks are actually just little scraps of JSX. And so if I drag, say, divider over here, it'll actually add the divider up here in my page. And so once you have this foundation of understanding the code, you can start to actually manipulate the code and bring some of that same feel of these no code builders onto your coded web app. Whereas you can also mess with the code if you want. And I'm not going to go too much further into some of the stuff you can build on top of this foundation. But just to show you a couple of things, as I had it set up here earlier, just having multiple views set up and being able to really see different views at once while doing your responsive design. And then also there's some AI set up. So it's like a natural way to deal with some AI. Make the background green if you want to change the color or something like that, you can give some AI prompts and it'll update your code right there for you. So with this kind of setup, you can change. And then the other thing you can do is you can actually make commits. And so if we look at all the code changes that have been done, this is very different from the typical code generation experience that I think many of us have maybe not suffered through for long. But looking at some of these older ways of doing this, getting a big chunk of generated code is no fun because this devtool is so synchronized with your JSX and understand your JSX so well and your style so well. When it makes code updates, it can actually do it in a very targeted way, very clean, surgical updates to your code based on just the things that you changed. That's kind of just basically run through of what mighty meld and how it feels. And now let's get into the interesting stuff around how this actually works. So I think, yeah, I can zoom in here. Yeah, so let me sort of walk through now. I can't. There we go. Let's walk through the basic setup of how this works. So Sweb app is the app that I'm running here. The envoy is actually this thing that runs locally. So the envoy is this thing that basically sits next to your app on your local machine or wherever your development environment is, and basically runs your app. And so when envoy runs your app, it does a few different things to it. It runs it by executing that command that I showed in the mighty melt JSon. It also instruments it so that when in the Dom, we actually can see where things come from in code. And it also imports a JavaScript package that basically provides this thing called the browser API. And so basically fundamental to the way this works. And this kind of, I think, gets back to that question of open standards. How do you deal with everyone running things in so many different ways on the web and the web being an open platform? And the answer to that is, well, we let you run your app the way you do. This works with next, this works with vite, it works with a bunch of other things, and that we use sort of things like a plugin and Javascript to augment that app when it runs. And so we can actually have some very standardized functionality in that app, even though you can kind of run it in a myriad of different ways. And so that's what's responsible for this app kind of being run and with the necessary things to talk to studio. And studio is this piece right here, which is, I guess, depicted over there. Right. And that's basically this interactive environment. It uses iframes to load the web app up and then basically uses the browser API to talk to that app. So if I click on something, it's actually talking to the browser API to understand what I'm clicking on and also to sort of get from the instrumented app, like where it is in code. And so with that information that allows studio to kind of run the app and allow me to click. Let's focus a little bit on the inspection experience before we get into changing and how things work. Just talking about the inspection experience, I click on something and because this browser API exists, I can know what I'm clicking on. And then the studio uses information provided by this thing called scribe to show this nice JSX tree and these nice styles. So let's talk a little bit about the scribe. What does the scribe do? Describe actually shares code updates. So basically it gets the code from the envoy. So envoy launches it gives not all the code, just code that you choose in your config file. It's the front end code essentially to the scribe, describe processes it, and then has these representations of the JSX entities and the components and the styles and everything and passes those to studio. And so studio kind of brings the two together. It has all this information coming from scribe, and then the browser API lets it know what you've been clicking on. And so that way you can see the JSX and also go up and down within a component as well. And so that's basically how studio shows you what you're clicking on, what you're working on in your live running app. And then the next big piece is change. So if I go in and do a change, how does that work exactly? Right. So I drag this up here and it updates on my screen. Also, I see it's updated in my diff. So how does that work? And that basically is a communication process where the studio basically tells the scribe what kind of move you want it to make. And then the scribe figures out what the new code needs to be and then passes the code update over to envoy and also passes updated representations to studio. So studio can actually update this tree. If I drag this, basically it gets a new tree from scribe and also updates the code and takes advantage of hot reload to make the app actually update. So yeah, new representation goes to studio and also new code goes to envoy, which triggers a hot reload which causes the app to update. And then you see it fresh in studio. And then as for the final piece in terms of being able to make commits, the clean code is here. And actually if I want, I can load it up in my own local GitHub or use vs code for the source control or however I do it traditionally. My email also has a few commands in to be able to make a commit. So you can make commits right here. And basically the way all that works is by just passing messages through. This part is pretty simple. Just passes a message through, describe scribe talks to envoy. Since the envoy is running on the local disk, you can execute git commands and things of that nature. And so basically these are how all the pieces come together in order to basically take the JSX and styles and understand them from reading the code and then pass that information to the studio so it can be manipulated. And then as you manipulate it, the updates then trigger updates all around here. And so that's essentially how you get a devtool that runs on the web, understands your code and allows you to inspect and kind of find your way, get your bearings in your apps, and also change visual things visually while of know. Oh yeah, I have one more picture about this. Yeah. And so this is kind of how the workflow works in the mighty meld universe. You can still do everything in vs code the traditional way, and mighty meld attaches to the visual code as well as little bits of the non visual code. And I think that's something that might grow over time. But looking here, like item name or like nav links, like basically showing you the actual data that you're working with, I kind of like to do this sometimes here where you can actually, you can see here movie category, right. I could change this to movie year. And you'll see year will appear twice, right? So there's like some kind of 2019 twice, some kind of interaction here with the non visual stuff and conditional rendering and things like that right now. And of course, sometimes you see things like that appearing in props and attributes. So right now, there's like a little bit of non visual code you can kind of see. But it's sort of thinking here, again, inspired by things like the unique game engine where they're just times where you're going to want to just be banging out a bunch of deeper code. And that's best done in text editors right now, which are also getting much better with all the new technology that's coming online. So that's basically how this visual dev workflow happens and how it seems like historically, browser devtools haven't understood your code, but it seems like the tides are starting to turn and it's both possible and seeming more likely that browser devtools for people who are building things and reacts and platforms like that are going to be much more code aware and kind of bringing all of us in the web community having that same kind of level of experience that unity devs have and unreal devs have, where you can build the thing, see it visually, move it around, run your app, pause the game in the middle, fix things, pick it up again, and basically really tie it into things. And it's a very exciting thing to be working on and also very exciting to be doing this on the web because the web actually has one extremely amazing thing that we've built, which is hot reload and unity. At least when I was working on it, didn't have hot reload, and so you couldn't really edit things on the fly. But because the web allows us HMR, new technologies around hot reload are coming out all the time. The web really just allows us to do this and work in a way that I actually think is even better than what you can achieve with things like unity. Anyway, thanks so much for your time. Glad to share this today. And yeah, if you have any thoughts or questions, my email is right here. Please don't be shy about reaching out. Thank you.