Hello everyone, welcome to my talk about reproducible and ephemeral development environments with Nix for our Golang projects. A little bit about myself my name is Haseeb Marjid. I'm a backend software engineer at Fintech called Curve. There's a link to my blog, a few fun facts about myself, very like cats, and I'm an avid village cricketer, bold and underlined on the village part there. But I'm looking forward to the cricket seasons dying in the next couple of weeks or so. It'll be good to get outside. So who is this talk aimed at? It's aimed at few groups of people, the first group being those who are kind of interested in Knicks and want to learn a bit more. And for sure we'll cover that. It's also aimed at those people who are looking to improve the developer experience, particularly around the consistency of the development environments. So things like going back to an old project like six months ago that you haven't touched and you're worried that you're going to spend a half a day yak shaving getting the development environment working, you're going to want to want to faff around with that. You want to just get on building features, bug fixes, adding tests, whatever you want to do, work on actual code. And so we'll take a look at how we can use nics to help with that. We'll also look at how we can make it easier to onboard new developers. So they have to type one or two commands just to get set up on a project. And especially if we can have consistency across lots of our projects, imagine at work then it means that developers kind of know they have one or two commands to write and they can jump between projects really easily and again get coding as fast as possible. Then also we don't just want things to work on our machine, we want them to work across everywhere we run this. So whether that be locally CI or even say like your various environments that you have production dev, etcetera, etcetera. And we don't want to annoy Samuel Jackson, do we want it to work everywhere? We don't want it to just work on our machine. So some of you probably wondering what's Nix? Nix is a declarative package manager, and we'll cover what declarative means in a second. But package manager in the sense it's a tool for installing packages on our machine, which is kind of, especially as a software developer, kind of a fundamental thing we need to do on our machines is install packages. It's similar to tools like ApT, Pac man or brew that you may be familiar with. It's powered by this thing that I'm going to call Nixlang. You may hear it called the Nix programming language and it's this pure, functional and lazily evaluated language. And what we mean by pure and functional is that it doesn't have really side effects. It does have like one or two, but basically for the same input, has the same output. It doesn't really depend on the state of your machine. So it means if we have these like NICs configuration files, we can kind of easily move them between devices and people because, you know, it's not really relying on the state of your machine, which is quite nice and lazily evaluated in the sense that it just works out kind of the bare minimum it needs. It's lazy, it's just like, okay, I have enough information to go build this package, I don't need any more. And that's particularly useful. You know, we have 80,000 packages in the Nix packages repository. We don't want to build all of them, for example. We just want to build the ones that we need for the packages we want, right? There's also this thing called Nixos Linux distribution that's powered by Nixlang and can be configured using Nixlang. And it's also powered by the Nix package manager. We're not really going to talk much about that. I do daily drive it, it's pretty cool. But just know it's separate from Nix. You could have like say an Ubuntu machine that's running the Nix package manager. So when we say something is declarative, what we mean is that we kind of just care about the final state of things. Whereas typically package managers are imperative where we're giving it like step by step instructions. So for example, if I was to say make me a cup of tea and then I gave you instructions like, you know, get the teabag, turn on the kettle, etc. Etcetera, that would be what you might classify as imperative. Whereas declarative is just me saying can you make me a cup of tea with milk and sugar? And then you kind of work out how to get to that final state. And it's very much the same with nics. In this case we have the sway tiling window manager that we want to turn on and we want to turn off the I three window manager. We're not telling Nix how to do that. We just say this is the final state and mix goes off, runs off and does that. One of the other cool things about generally when things are declarative is we often put them into code and then we get other benefits, like version control, easier for people to review, easier to reproduce as well, because again, it's not really caring about the state of our system. If I don't have I three installed, for example, Nix will just work this out fine, it just won't uninstall I three. It's pretty cool. Then I think, I think it's definitely a really useful feature of Nix. And then once I definitely will have converted you to Nix, you can start busting this out into conversation. By the way, I use Nix to your friends and just doesn't matter what the conversations, you can always change it and I'm sure they won't mind at all. What's the problem we're solving? Well, imagine like typically when we have binaries installed, we might have them in user local bin, like go Lang ci lint. There are a few problems that the typical package managers have. It's like, what dependencies does this need? Like runtime and a build time? Like how easy is it to discover that? What configuration flags was this tool built with an environment? Variables, right? If you wanted to like build the same version yourself, how'd you do that? And then how do we have two versions of this package? What if we need to have version two, for example, in version one? Now as far as I'm aware, Go Lang Ci lint doesn't have a version two, but it might. And for some projects, you might want to use version until you've upgraded it. And for some projects, you might want to use version two. How do you do that? Because typically our package managers replace the binaries in place, right? So we replace this with version two. So how do we maintain multiple versions? And there's various packaging solutions that kind of solve some of these problems. You have like snaps and flat packs which create these sandbox environments, and they have their dependencies, I believe, in these kind of sandbox environments, and they don't really interact with the rest of the system. We have ASDF for managing like multiple versions of some of our tooling, like Go node, Python, et cetera, et cetera. We have virtual environments and to a certain degree go modules, so we can have per project dependencies and we don't need them globally installed. More so for virtual environments than go modules, but yeah, virtual environments used in python. For those of you who may not know, to summarize this section, we want to have reproducible and ephemeral environments. NICS is an ecosystem of tools. So we have NICs, Nixos, Nix packages, the Nix programming language, the main thing, of course, being the package manager. And then our current packaging systems all have various flaws. There's nothing in software engineering that's a silver bullet, Nics included. It can be a bit complicated. Nics, you kind of have to learn this programming language, which puts people off. But I think the upsides beat the downsides of Nics, in my opinion. So if we take a look at this demo, we look for this go lang ci lint binary, can't find it. We go into this project, we load this dev environment, and then when we look for this binary, we can find it. And when we leave this project, we will no longer have this. And this is kind of the state we want to get to. This is kind of what we want for our developers. And we want to make sure that people are getting the same versions of tooling as everyone else. In this case, I think it's version 1.56.2. We want to make sure all the developers are getting that same versioning. And when I say ephemeral development environments, what I mean is short lived, temporary in the sense that just existing for the lifecycle of this project. And when we leave it, the environment's kind of gone or not loaded. But yeah, so that's kind of what we're going to achieve and we're going to take a look throughout this talk how we can achieve that. So how does this relate to go lang? Well, with go, we need tooling to aid development, right? Like we might need binaries to generate code coverage reports. We may have tools to vet our code. Static code analysis, container code analysis, all these things that go into development, you know, Docker and Docker compose. We have, you know, dependencies for our projects and we're not really going to worry about those, you know, that we manage viago, but the other tooling we need to aid our development, maybe we have a task runner, like we have makefiles, right? Like we might do make lint or make test. And we want to make sure developers have similar versions or the same versions if we can. And we want to make sure the same versions are running in CI, they're running locally, because I'm sure we've all been bit by the bug where it fails in CI is working locally and it's just because of a version mismatch. Couldn't see our word for a second. So one way we can kind of do this is, and we do the set curve for some of our projects is we have this tools go file. And essentially what we're going to do here is we're going to manage our dependencies using go module. So we install these various packages using go modules and then we add these underscores here to trick go modules into thinking it's important. So if you do a go mod tidy, it doesn't remove these. And the cool thing is then they're all kind of managed Vigo modules and we can do an update. We just update one file and it will update our dependencies and etc etcetera. It's quite nice. And we have this build flag so it doesn't get built with our binary and that works for go dependencies. And we maybe have a make target like this which installs these dependencies in our go path bin folder, which is not which works. But then we encounter similar problems as what we're talking about before. What if between projects we have version one of one tool and version two of another tool. We kind of have to remember then to run this tool which is going to overwrite the binary in that go path bin folder. There's something else we have to remember to do when we're jumping between projects. And then what if we want to manage tooling not related to go? What if we wanted to make sure that the user has GNU make or GNU parallels or some other CLI tools and maybe we have some bash scripts or something. And we want to make sure certain tooling is available in that. And so I think this is where Nix can come in and fix a few of our problems. So let's take a look at how we can create a development environment in Nixon. So imagine we have a project like this, really simple go project. How are we going to create this Nix environment? Don't worry about the syntax of nics and what's kind of happening behind the scenes. We're going to take a look at that just later in this talk in a couple of minutes. But first we're going to take a look at how we can create this development environment. So we have this flake Nix file. Think of it as a main go file, as the entry point to our Nix configuration. It has a bunch of inputs and a bunch of outputs. In this case, our inputs are all basically going to be git repositories. So one of them being Nix packages, which is this repository that has 80,000 packages. So this is where we're going to install our packages from, has a bunch of outputs. Nix flakes can output a lot of stuff. They can output build a docker image they can build an ISO, they can create a development shell, which is what we're going to do, Dev Shell. They can build a package, they can do lots of various different things, but we're really just going to focus on dev shells today. But it's just good to know generally speaking. So we have this helper library called flake utils, which basically just reduces some boilerplate in our flake where obviously packages have to be built for specific architecture. So you know, like x 86 64 Linux or Arch Darwin, you know, it's like AMD and intel versus arm based chips, different architecture. So we have to build the binaries differently and so packages are built, we have different slight packages. So we're here basically what we're doing is we're just specifying that we want to just get the packages for our system architecture and flake utils. Lib is this library that helps us reduce the boilerplate to do that. But that's basically all we're doing here. So don't necessarily worry about that here. What we're doing is we're creating a default dev shell. We have this packages make shell function. So between the kind of curly braces is this function and we're passing a parameter called packages, and these are all the packages we want to make available to the user of this dev shell. The details again don't matter too much that I just took that from a project I had called Optinx, which I've linked later on. You can take a look at that. So we take a look at, so how do we use this? Well, if we kind of look for the binary, we won't find it, it's fine. Then we do nixdevelop, which will load our development shell that we just created there. And then when we look for our go lang ci link binary, we can kind of see at this funny path next door, some funny characters hash maybe, and then yeah, cool. So we've created a development environment. So if we kind of summarize what we've done so far, well, we can leverage flakes and dev shells for installing packages. We can load into those shells or that shell using nixdevelop. We can make sure each developer gets the same package. We have this concept of this flake lock file which locks our inputs, and we'll take a look at the syntax of that and how that works a little bit later in this talk, but just keep that in mind and we can update this lock file, but we have to kind of manually do it. So one other thing that's quite cool is we can use this again nix agnostic tool called diranv. And with Diram what we do is create an MVC file, and what we put in this MVC file will get executed when we load into this directory automatically. So what we can do here is do this useflake, which is kind of this helper function for running Nix developer automatically. The very first time we load into a directory that has deriv, we do have to has this MVC file with Dirham, we do have to approve it so we don't just run arbitrary code on our machine. So let's take a look at what that looks like. So imagine I'm at work and I need to add a feature to a project. I clone this example project, I try and find this linter. I want to lint the code right, can't find it? Fine, I load in. I do Durin Valalau, because remember that first time we have to do Durin Valalau which has this useflake. It will load in this dev shell. In this case it's ready cached. It doesn't need to do anything. It can just load in one that I already had. Then in my case, because I'm using starship prompt, and I'm sure other prompts do this as well, it will let me know that I'm in a development environment here with the viya, and it has that little flake, which is quite nice. Just a good reminder that you're in this dev shell. Then when we look for this binary, we can find it. When we leave this folder, we can't find anymore. And so that's again that ephemeral nature I was talking about. And one of the cool things about Dirham is the first time you go into a directory, it'll tell you you have to do Durham Valao, so it's not something that you have to remember, it will tell you. And again, you only have to do it once. Next time I go to that directory, you won't have to run that again. One other thing we can do with Durham, the Nix flakes, is we can point to a dev shell or some Nix configuration that has a dev shell in a remote repository. In this case it's on GitHub, and we can use that if we want. So we can share configuration, and we could use multiple flakes as well if we wanted to. But we're not really going to get into that for this talk. But just to know if you want to have a remote development environment, you can as well, or the config remote, you can use that. One other thing that I think can really improve the developer experience and something we can manage with NICs is pre commit. So you know, we have these things called githooks, which is these scripts we can run at various stages of the git process, like pre commit, post commit, pre push, post push, etc. Etcetera. Then there's this tool which can be a bit confusing, as in this bit confusingly named called pre commit, which will basically help us create these pre commit git hooks for us. So we can create this using nics. So if we do pre commit hooks, add that as input. Again, you can name these inputs wherever you want, just helps to kind of make them somewhat related. We add that into our output section here and say we want to create these pre commit hooks. It has some built in hooks for go that we can use. So we use golangs. We're going to enable the linter and we're going to enable tests. The cool thing is it will only lint and run tests on the files that have changed, as in the ones we're trying to commit. And so yeah, we get some really fast feedback when these run. Saves us time waiting for CI and save some credits as well, build time that could go used for somewhere else. Then we can add the to our make shell function that we had. And the shell hook is just a command that will run automatically. When we do nix develop, I will load into the shell, which is going to happen automatically when we're using Diran. So essentially when we go into the folder it's going to install our pre commit hooks for us. Whereas normally the developer would have to remember to run the pre commit install like it's another command they have to do and now they don't have to think about, which I think is pretty cool. And with these pre commit hooks you can get some again really fast feedback. So it closes that kind of feedback loop and lets the developer know something's going wrong or not. To kind of summarize this bit, what we've done, we can use Durham to further reduce cognitive load on our developers. We can use flakes from remote git repositories, share them between multiple projects if we want. We can also manage precommit in nics. Just something to note that pre commit is usually managed using a YAML file, and now we're using a nIcs. And some people do have an issue with abstracting away from the original the way we configure a tool. I don't mind it, but just something to consider. It's kind of the next section I want to cover is how does Nix work? Like what's happening behind the, behind the scenes. So everything in Nics is an expression which I believe is quite common for functional programming languages. Remember, this is powered by Nixlang. And so what we have is we have a file, maybe it's called shell Nix, and this will get imported somewhere. And we have this function essentially here in this file that takes in one parameter called packages, and then we have these triple dots which ignore any other parameters passed. Then we have this function call called packages make shell, and we pass a bunch of packages we want to install. So in this case, this nix expression, we return one nics expression from the file, which can be a compound of other Nics expressions put together, but we always return one. So in this case we're returning this function, and again this, this file will get imported and during the import that you'll have to pass packages. And yeah, we were kind of doing this with our flake dot nix file, but it was a little bit more hidden, I guess, what was going on. But that's kind of what nix is and kind of what we're doing here. So you might be wondering like, okay, we're, what's this go Lang ci lint? I get it's coming from Nics packages, but what, what does that mean? Well, on nix packages, the GitHub repository, we can go find the go Lang ci Linux expression, and it has this function called build gomodule, which is a helper function for building go modules. And you can see as like a name a version where to fetch it from. GitHub has a bunch of other information about how to build it and if it has any dependencies, et cetera, et cetera. So we have this nix expression there. Cool. If we dive a bit deeper and look at what's behind the build go module, it abstracts away the standard env derivation, where this derivation function is the most important built in Nics function. So when we're building packages, what's actually happening? It's a two step process. So when we do nix develop behind the scenes it'll be calling like Nix build of some kind. And behind the scenes we'll be doing this, this in two steps. And you'll see why we do this in two steps. So the first step is evaluation time. We take the nix expressions and the Golang Ci lint expression, and we return a derivation set. This DRV file where a derivation set or derivation is just a set of instructions how to build a package, kind of like a recipe. Then we have this build time. The derivation is built into a package, and that is what has a side effect on our machine where stuff is actually getting installed. So let's take a look at derivation. So derivations are put into our next store folder. They have the format hash name version dRv, where a hash is a cryptographic hash of all the inputs to that derivation. So let's say we have this go 121 eight derivation, even if we're building go 121 eight, let's say we change an environment variable. We are going to get a different cryptographic hash there, and so we're going to get a different derivation, and for all intents and purposes it's a different package as far as Nix is concerned. So derivations and also packages are mutable. I mean, you can obviously go change them if you wanted to. Nix discourages you from doing that. You probably shouldn't, but you can if you really want to. Let's take a look at what this derivation looks like, and we can run this command at the top there. Don't worry if some of this is cut off. Doesn't really matter, it's more just high level what's going on. So we have a builder, how we're going to build it bash we have a bunch of environment variables. These are the only environment variables made available during the build. We have a bunch of input derivations, so other derivations this depends on, and Nix will make sure these derivations are built into packages before then we have a bunch of metadata where this package is going to get installed, what system it's for, name of the package. Then we have this package, which again is immutable. So if we wanted a different version of this package, let's say environment variable changes, it would be at a different path. And you can think of the path as a unique identifier, as far as Nix is concerned, of a package. Then there's some sim linking done later, which will determine which binary we end up using. Even if you think they're basically the same. Then within Nixdor we have everything we need. We have the binary, we have Sharego and has a bunch of other stuff, which is quite nice because our packages are immutable and we can kind of pre compute them. So Nix has a bunch of servers available to it that are kind of pre building these binaries and packages. And so what that means if a derivation says we need to build a package at this path, we can check if that package exists in our path. If it doesn't. We can go fetch from various different caches. In this case I'm fetching from the official Nixos cache. And the cool thing about that is often because these are being pre built and you can pre build them yourself as well if you want, and you can pre build your own packages if you wanted. It just means we just have to fetch them and download them. We don't actually have to build them ourselves on our machine, which is really cool. So often we're just downloading stuff we're not actually building because lots of these packages are pre built. Another advantage that this kind of approach gives us is that the dependency tree is explicit and we can see what go depends on and then what those tools depend on. So like it depends on bash, and then bash depends on glib C. And every anyone with this exact same unique identifiers and you know, starts with k, seven, ch, et cetera, et cetera will have the exact same dependencies as us if we print out this dependency tree and nextore basically becomes this kind of graph database of our dependencies. One other thing that doesn't matter so much for dev, specifically for dev environments, but it's kind of useful to know because of this approach of we're not updating stuff in place. We can kind of have this concept of generations and profiles and we can roll back to earlier versions of generations. We can also then have atomic updates if we want. Sorry, we do have atomic updates, and that basically means that if something fails during the update we can go back to an older version. Or we don't even end up updating the NICs profile at all because these binaries just end up getting mapped to stuff in Nick store. And yeah, unless we garbage collecting cleanup stuff it's kind of going to be there. One other thing to note is you'll notice that we use an epoch time of one, that is 1 January 1970. The reason for that is because the timestamp date time can be a form of non determinism, because sometimes that gets injected into the binary. What we want to make sure is that we build the exact same binary. Otherwise every time you built a package you get a different binary and so wouldn't, you know, we don't want that. So we set this to 1970 and we make it a deterministic timestamp. Then in the Nix world this is what bincat looks like, not user bincat, but in NICs store. Just to kind of summarize, NICs derivations allow us to have immutable packages, require us to make our dependencies explicit. One thing to kind of note is if a package is not nix packages, you will likely have to package it yourself. But because Nix packages has 80,000 other packages, often or always, I found if I need to package something, it's just that one thing. The dependencies it needs is almost always in nix packages itself. The next bit we're going to cover is nix flakes. So how does Nix flakes relate to stuff? So nixflakes basically take state on our system and kind of put them into code in the sense that we have these things called nix channels which would refer to what version of nics packages were pointing to. And what we do is we just take that and make that kind of more explicit and we'll see exactly what that looks like. With this flake lock file, we lock our stuff to specific revisions, and that means other, because we can put that lock file in code, other developers can then point to the same inputs. We can also use other git repositories, non nix related, and manage them using our flake nix file if we want. They also define some basic structure, because now our flake Nix file becomes the entry point for our Nix configuration. So you know, that's kind of the first file you can go to to look for that Nix configuration. So as we said before, we have a bunch of inputs, a bunch of outputs, the main input being nix packages, or the default input we get when we initialize our flake. Then we also generate this flake lock file, which has this kind of concept. So this is the lock for the next packages input. We have this null hash, which is just a hash of the contents of that input. Less important because it's a git repository, but we could have non git related inputs. In this case, we also have a revision. So anyone using this flake Nixonflake lock file combination will be locked to this revision until we update that lock file. And that means if we're using this in CI or other developers using this will be point to the same version of Nix packages, which is really cool. We can also use like say a GitHub action or CI to update this flake lock file for us if we want to think a bit like dependable style stuff, if you want to do that. To kind of summarize nics, flakes improve reproducibility across our system by locking our dependencies. They provide a more standard way to configure our system. But do note they are an experimental feature, nics, and they could break in terms of like they could be breaking changes, so just keep that in mind. But I think they're great and I use them in all my projects and my own Nix configuration. So kind of one of the final topics we're going to touch on is CI. We've spoken about we want consistent environments between what's running locally and what's running in CI. How can we do that? And definitely we want to kind of leverage Nix's cacheability and sharing dependencies. GitHub Actions has some great stuff, especially from I think determinant systems for leveraging caches and various things to speed up your pipeline. I use GitLab CI and I found this great project by this user called Cynerd and it has this next job. And what it does by default is leverages the GitLab cache, but we could use a cache from an SSh machine as well, or ssh to a machine and copy from there. But essentially what it does is before and after the job, it will just copy from the cache to the next door of the job and then from the next door to the cache. So it shares dependencies between different stages and different jobs. So in our GitLab Ci file we include this GitLab Ci file, the GitLab project we just saw. Then I have a stage called pre that extends this nix job and we just do nix develop to install our dependencies. Then at future stage dependencies have been cached, they'll be copied over to this job and we can do nix develop C Go Lang ci lint run, whereas normally we'd be in that NaICs development environment and we could just do Golang Cilantro and we should be running the exact same versions because of the nature of Flake Nix and the various things we've talked about with Nix. So that's really cool. And this is kind of what it looks like, eleven and a half minutes total runtime. I think before I did this change it was 22 minutes, rather anecdotal, but something to note. Kind of just taking a look at Ci logs again, we can see it's copying from that cache, which is really nice as well. And this was definitely me with Nix when I first heard about it, I didn't really get it, and then eventually it clicked and I think it's really, really cool. And hopefully you guys either think that at the end of this talk, which means I've done an amazing job. Unlikely, but I'd recommend just giving it a go and seeing what you think. But before we close out this talk, I'm sure some of you have been thinking, or maybe even screaming at your computer, why not Docker? And I definitely was confused, like how does this relate to Docker? So remember, we're talking specifically for Docker in terms of like as a development environment. I think Docker is still great for building and packaging and deploying our stuff, especially on the cloud. And specifically what I'm talking about is Docker files to build docker images. And one of the problems Docker has, I think by being imperative, you'll often hear the term it's repeatable, not reproducible, often because of the package managers we're using. But imagine you have a Docker file. If two people try and build it like six months apart, you're probably going to get a different docker image and therefore it's not reproducible. One other problem I have with, say, Docker dev containers, which versus code plugin, which I think now works with jetbrains as well to try and make it easier to develop within a container, is what if I have specific tooling that I want, I now need to make that available, like say fzf z oxide. I have a specific shell, I have to make that available in the Docker image, and I'm potentially bloating that Docker image as well. I did find a way to personalize it. You can kind of run this startup script, but that was a bit slow and a bit more cumbersome than I found comparing it with nics dev containers. Sorry, Nick's dev shells. So just something to note there. Give both a go, see which one you prefer. But I think I found nics a lot easier. A few things you can look at in your own time to further use nics even more like nics. All the things here's a link to my slides. I'm going to have a bunch of other links, just go through them in your own time. Just a bunch of lots of literature and articles and YouTube videos. I want to give a shout out to Vimjoyer specifically. They do really great videos on YouTube. Highly recommend checking them out. And just a thanks to everyone who gave me feedback on this talk and improved a lot better than it was, I think version one. And thanks to comp 42 for giving me the chance to talk about this. I really like talking about Nix and want to share it with other people to see let them know what they can do. And thanks to you of course, for seeing through this talk and listening to me ramble on about NyX. Hopefully you found that useful and you'll give Nick's a try. Thank you very much. Have a lovely day and enjoy the rest of the conference.