Hello there. Good morning, good afternoon, good evening. Whenever you may be watching this, I'm excited to speak to you today on my talk called nuclear rust fission powered code for the modern world. I am John Darrington. I work for the Idaho National Laboratory. Here in Idaho, we have a laboratory. The land we own is roughly the size of Rhode island. We were behind the first city in America powered with nuclear power. That was Arco, Idaho, and we have been working for a long time with nuclear energy and nuclear fuel, seeing what behaves, what misbehaves, and working in that realm for a very long time. And I'm excited to talk to you today about our experience with the rust programming language and some of the reasons why we chose rust and we went that direction. So just a little bit about myself. My name is John Darrington again. I'm the lead architect for the digital engineering department at the Idaho Idaho National Laboratory, work primarily with data management and work in the digital twin and digital transformation spaces. I've also been working primarily on a data management warehouse, or data lake, more of data Lake at this point, called deep links, and other various tools in that ecosystem. I've been a software developer for over ten years, been working in that realm, as well as just working with data management overall. So let's get started with the fun stuff. This is the preliminaries we have to get this out of the way. But let's start talking about the fun, or in this case, not so fun. If you were watching the news back in 2011, you would maybe recognize this image, and the scale of it might be a little hard to see. You can see there's a truck off here to the left here, a big crane truck. This is one of the outbuildings of the. Or not the outbuildings. One of the buildings of the Fukushima power plant. And in 2011, a 15 meters high tsunami disabled the power supply on three of the Dai Chi reactors. All three of them largely melted in the first three days, and it took until December to reach an official shutdown condition. Now, why do I start with a nuclear accident, so to speak, for talking about rust and talking about our experience with rust? Well, I think it's a little bit easy. So no industry is immune from accidents, but all industries learn from them. There have been three major accidents in the nuclear world that we kind of refer back to three Mile Island, Fukushima, and, of course, Chernobyl. In every industry, there are accidents, and hopefully we learn from those accidents. And programming, to be honest, is more than an industry. It affects all these other industries. In and of itself, and hopefully we learn from them. And I think that's kind of where we get with rust. But let me talk a little bit about where our department and where inl kind of finds itself in the nuclear industry. So this is the state of software industry in the nuclear world. We are very risk averse for obvious reasons. I mean, it makes sense. We're dealing with nuclear power, nuclear energy, things that could go very spectacularly awry, as we've seen with Chernobyl, and affect millions of people and cause untold damage. So we are very risk averse in this industry. We want to make sure that what we're doing is safe. We want to make sure that it obviously comes in with budget, but mainly it has to do a lot with safety and making sure that we're doing the right things. Because of that change is slow. We're so adverse to risk that we worry about changing things too rapidly and moving too far forward. And so that kind of change is slow. It has to go through committees. It has to go through the nuclear regulatory committee for even running simple reactors. It takes a while to get things moving. And in the same vein, again, committees, everything's done by committee. It feels like we have committees for software and cyber control. And it's not bad things. That's what I hope to get through. It's not necessarily that they're bad, but everyone in the committee has their own goals, has their own experiences, and that can cause things to be extremely slow. Again, risk averse change is slow. We find ourselves in that industry, in just the software that we're dealing with. And there's a very set way of doing things. We've done this for years. It's been safe. We're not going to change it. We're using this same exact microcontroller. Weve used it for 20 years. Nothing's happened. It must be safe. We're going to continue to use it. And it's hard to push up against that because you're trying to introduced either new technologies, new paradigms, and if something's worked for 20 years, that doesn't necessarily mean it's better. It just means it's worked for 20 years. And who knows that there might be a catastrophic failure in the future. But again, it's a set way of doing things. And we have to push up against that. Ancient protocols. We are dealing with ancient network protocols, ancient communication protocols between microcontrollers and facilities. And so we have to find technologies and things that work in these ancient setups. So again, this is where we find ourselves in the industry. So this is the original slide that I put together in 2022 about why we felt the need to introduce a new programming language, especially in the department. At inL, for one, we have our deep links data warehouse. We were using a high level programming language and we needed lower level programming language than we typically use. We were looking at embedding it or using either webassembly or putting it closer edge compute. We weve using a lot of node js, and that's really not good for long term use by a department, mainly because it has some performance issues. Again, it's cpu bound, a single threaded great async for I o operations. But if you're doing complex computations, no chances is definitely not the right tool. And so again, that leads to a lack of well adopted scientific computing, at least at the lab. And runtime requirements. We have a lot of python, and while in use, again it has a runtime, so it's not available for all deployment pathways, especially on microcontrollers, or on things that have either need to be very lightweight like DAs data acquisition systems, or things that can't have Python's bloated runtime on top of it. And yes there's Python, and yes there's whole bunches of directions we can go there, but those are complex. And trying to get data scientists to use that can be really difficult. Performance is a concern for issues not covered by internal c libraries. I think we've all seen that meme with the really high performance speed car and says Python. And then it zooms the picture out and it's Python on top of this tow truck. That's C and C Plus plus. And that's kind of what we're talking about. And then again, Python is extremely permissive and allows the easy introduced of bugs. And if anyone's worked with either university code or code coming in from data scientists, they're not software engineers. Well, even software engineers make mistakes, but these people just want to make sure something works and so they'll get it working. And regardless of whether it's going to have bugs or not, and the language doesn't care, it's not going to try to make them do that. We had to make a conscious decision that's a constant in any industry. You have to make decisions before they're made for you. We looked at C Plus plus, but the tooling is difficult to learn and can cause issues. And then we were looking for safety and lower weve languages are necessary for operational control. Why are we looking at operational control? Well, that's a good question. And the answer is, because we are trying to build digital twin. And a digital twin is basically a digital representation of a physical asset that eventually, and you kind of can't see the level seven here. It's a little cut off by my face, but the level seven is artificial intelligence and automated design optimization. And so weve trying to build this digital representation of a physical asset that would eventually control the physical asset or have some kind of a feedback loop back into the physical asset. And so if we're trying to build digital twins of nuclear reactors, well, then we need to deal with safe and performant languages, because we are dealing with, again, nuclear reactors and trying to build these digital twins. And again, you can go through a little bit and look at these different levels of digital twins as you go through. Don't worry too much about that. Just understand, again, it's a digital asset or a digital representation of a physical asset that will eventually, it can predict things about the physical asset. It can interact with the physical asset. It has feedback loops with the physical asset. And so it can have a lot of those combinations and changes. Roughly 70%. And this is why safety matters. And I took this from otka.com. 70% of all cves at Microsoft are memory safety issues. Two thirds of Linux kernel security vulnerabilities come from memory safety issues. Android has a bunch of vulnerabilities that are safety issues. An analysis of that, my favorite one is this last one of zero days that were discovered being exploited in the wild, found that more than 80% of the exploited vulnerabilities were because of memory safety issues. And so if our stick was dangerous security issues and vulnerabilities in control software or just bad run states or, sorry, bad states in the piece of software, then our carrot was memory safety was, hey, we can potentially get rid of an entire class of bugs if we can find some way to ensure memory safety for our software programs and for what we're trying to build. And so this led us to one obvious conclusion. And I mean, you're listening to the rust conference, so obviously I think you're coming to the same conclusion that we came into. We put together a few different languages and kind of pitted them against each other. And the one that came out to be a clear winner for our use case was Rust. Now, again, this is the original slide from 2022, so not all this might be correct. I wanted to maintain my original look at Rust for the department and the area and the Idaho national lab so that you could get a glimpse into exactly what we were looking at and what we thought about it before we even started using it. And without having spent tons of time researching it, I talked about the idea that it was sponsored by Mozilla because we were concerned about long term usage and support. Again, statically typed and compiled language. This part was really important to us. Can target a lot of different chipsets and different oss. You can even in some cases run it without an operating system as long as you have a memory allocator. So it made sense, the no garbage collector and the compiler enforcing that memory safety. Again, the carrot and the stick, and our carrot was memory safety. I could have made an entire few other slides on this if I wanted to and talk about memory safety. I really appreciated the lean standard library large package ecosystem with a centralized package manager. That's one of the reasons we didn't want to go with C. Why there are package managers for C. There is nothing as standardized it feels like to me, and I might be wrong. Again, this is the original slide, but this one just jumps out at you. Concurrency model built in now, we did talk a little bit about its complex and much steeper learning curve, and we'll talk a little bit about our experience with it. But with complexity comes powerful features. The reason it's so complex is because it does have powerful features, generics, abstractions, pattern matching, trait system, macros. So, I mean, there's a lot of power behind it, and I wanted to make sure that we understood that, yes, it's complex, yes, it has a steeper learning curve, but it has a lot of powerful features. We talked a lot about the community. We care a lot about our developers here at INL. So we want to make sure that they feel like it's an easy setup and an easy way to get working. And probably the most important piece on here for me at the time was the interop with other languages, and I say excellent. And again, this is the original slide from 2022. Now that we have actually introduced Rust with Python C sharp node, I can tell you it's not bad. It could be better, it could be worse. But in a lot of cases, there are powerful libraries out there already to make these interops a lot easier to deal with. I've successfully written various Python modules and node modules and even a C sharp package in rust under the hood. And so it is possible it does work. And we've done a lot of work in Rust since we started this in 2022 1st kind of our bread and butter of the department is the deep links open source data lake. We have time series and tabular data support in there. Originally it was almost pure node js, and you can already see a problem with dealing with data lakes with an ETL pipeline and doing node js. They wanted to unify the languages at the lab in use at this place, and node was kind of picked because they had to do a front end and they wanted to make sure that it was a full stack developer. So we started to use a lot of node JS, and originally it was just pure node js. We introduced Rust three months after that initial slide and initial discussions. And what we introduced was a postgres fast loader module for tabular and time series data to more quickly load that into a timescaledb backed postgres cluster. And it was patterned after TimescaledB's own concurrent postgres loader. We were able to write this in Rust and have it implemented as a native node module. So we were able to use it in the same code base without having to stand up a separate service or proxying it or figuring out anything like that. It just natively works out of the box. Right now we are in a massive refactor of deep links where Rust is going to drive our new tabular and time series data support, and we are integrating with a technology called Delta Lake. If you haven't heard of Delta Lake, I highly recommend you look it up. It basically allows acid transactions on hard storage for time series and tabular data like Parquet files. Just acid transactions and parquet files on hard storage. It's fantastic. And we're using Rust to drive that. We are working on backing our open source graph technology with Redis graph in order to make that fast. And we have a loader that will load snapshots of the graph at any point in time into redis. And weve written that in rust again, and we are looking at integrating a large language model, particularly llama rs if you can look it up. And it's the rewrite of llama C in rust, and we are looking at integrating it again into Deeplinks'core code base. And all of this is possible because of Rust. We've done several other things. One is a program called Serval. It was a websocket multiplayer game server used to fuel or not fuel, used to back our multiplayer game sessions for our digital twin. So we use uni a lot in house to represent these models in a 3d space and let users interact with them using Microsoft HoloLens using other tools. And so we wanted to let users interact with other users. And so we've created this websocket multiplayer game server to make that a reality again. Drives the AR VR multiplayer integrations as well as the multiplayer integration for our deep links viewer. For personally was creating a deep links loader which is both a Python and a c sharp module for caching time series and tabular data from Deeplinks locally. Letting data scientists work with deep links in Python in a more easy fashion. Letting us pull it down into a duckdb database to let them query with SQL to their heart's content without having to slam the data lake with requests over and over and again. Leverages DuckDB and right now we use it in a lot of our machine learning and our artificial intelligence pipelines here at ino. And again, none of this would be possible without Rust. This year we finally hit the big time in dealing with Rust. We finally had the opportunity to do what we wanted to do since we started discussing Rust and why we wanted to use Rust. This is the AGN 201 nuclear reactor. It is rated for producing five watts of electricity. It is hosted at the Idaho State University in Pocatello, Idaho. And we, Idaho, Idaho, Idaho National Laboratory, were asked to create a digital twin of this nuclear reactor. Now, if you remember back to the levels of the digital twin, we have things like real time analytics integrations and then eventually a feedback loop into this reactor. Now, while we did not complete the full feedback loop because of nuclear regulatory committee regulations and rules, we were able to basically get all the way up to that point and demonstrate that we could in fact, potentially control the reactors with this safe program. And so we built a system with deep links at its core and a lot of rust. So we have down in this operations portion here, we have to get the live asset data from the nuclear reactor. We wrote a program called Jester. It is a rust file watcher and data acquisition tool for pulling in data from data acquisition systems, reading it in and sending it off to the Deeplinks data link. And again, that was written in rust. We had a very tiny machine with tiny specifications to work with. This is not a very powerful machine. And so Rust allowed us to make something that was performant and that could fit in the resource constraints that we had and then be able to send that over the network to the Deeplinks data lake. That data lake drove machine learning and artificial intelligence. We passed that data out into basically Jupyter notebooks, and these Jupyter notebooks are using that deep links loader that I talked about for pulling in time series data from deep links in a better manner than just constantly hitting the database. And then we built a program around paper mill and around others to control the execution of Jupyter notebooks and the code that our data scientists put together to run anomaly detection, to run all sorts of different algorithms on this data, to see all sorts of different things about this reactor. And again, this is all facilitated by the use of rust. And then that data gets shoved back into deep links, and it goes out into this area here, where we have a live model and human interaction. We have an AR and VR headset that is, again, using the same deep links loader technology, but as a C sharp module written in Rust to drive our AR VR, and we have that running on a HoloLens headset. So, again, Rust was able to target this and work on HoloLens. It was fantastic. It made everything easy. I didn't have to write, or we didn't have to write the code five times in five different languages to make it work. We were able to maintain a much smaller workforce to accomplish this main goal. Our bet paid off. So we launched the AGN 201 digital twin in less than six months of development time. Now, granted, we had a lot of the work done with the data lake, but again, we still had to write the data acquisition system, we had to write the machine learning portion, we had to write the loader, and we were able to do that in six months. Now, this does not include the rust ramp up time, and I know that is a concern of a lot of developers, but I have seen that, in all honesty, people pick up rust fairly quickly when they are motivated and they have a problem to solve. You can't expect your developers, and you can't expect those that work for you to pick up rust in a weekend, or expect them to pick it up when you don't have, or when you haven't handed them problems to solve and things to work on. We maintain speed and accuracy of the data by using native modules in Rust. So the speed of getting that data off the data acquisition system into machine learning and then back to the end user was extremely important. And then the accuracy of that data that we had no drops, or that we were able to maintain those drops and buffer and send, and we were able to accomplish all of that with Rust. And then this one is my favorite. I believe that we demonstrated the ability for rust to eventually become part of nuclear control instruments and paradigms. We now have a working digital twin that is functional accurate, safe, and hopefully we can demonstrate that. Hey, we have new technology that we should utilize in dealing with nuclear reactors. Lessons learned. The big one is, again, find what drives management and those that make the big decisions and use those to drive your adoption of rust. Drive your innovation. Have a game plan, but leave wiggle room for all the people that you have working for you working in rust to experiment and try their own things, and again, give them a problem to solve. Let them prototype early and quickly and often, and you'll be amazed about what you can develop and what you can create with Rust. My experience in dealing with rust in the nuclear world has been nothing but positive. There are hitches. There are little ugly spots. We talk about the asynchronous implementation in Rust that could use some work. We talk about libraries that are rough to use. We talk about the trace system, which can be difficult to understand sometimes, and proc macros. But we have so many good tools now. We have so many good resources that it doesn't, again, give them problems to solve. And you'll see your individual developers and those working in Rust just continue to excel and continue to use this language in a powerful way. Thank you so much for the opportunity to talk to you today. I really hope that you enjoyed this presentation. I really hope it's encouraged you to pick up Rust, see what it can do for you, because I promise you, in whether it's 510, 15 years from now, Rust will be involved in nuclear energy and it will be making the world a safer place.