Hello everyone and welcome to my talk multiple terraform projects in a Monorepo and today we will try to answer a question, how to survive that mess. A couple words about me. I am a DevOps engineer and I used to be a software engineer. I try to automate things and I fan of clean code and open source and infrastructure as a code. Here you can find QR code and link to my profile on LinkedIn. So if you have questions about this terrap compose tool, or if you want to just talk with me about some DevOps topics or something else, you can reach me out there. So let's start when someone tell you that it has everything as an infrastructure, as a code, it might be something similar to that picture. So it probably not something that you are expecting out of these words, right? So you can see there as a code, but it may be copy pasted, or maybe not clean, or maybe not updated and so on. So it might be different expectations out of it. I was in this situation several times and today I will try to guide you through my experience out of managing such projects. And our agenda for today is first I will explain prerequisites that we have. Then we will have a problem deep dive. Then I will try to explore existing solutions. What we have there, I will introduce terracompose then and a short demo and a couple final words in the end. So let's start. We have as a prerequisite a set of projects in a monorepo. They have various workspaces, which is mostly equal to environments, various structures and terraform versions. And if we look into our projects, you might have similar situation that you have some projects in your folder, whatever name I select projects. And you might also have some common stuff like creating vpcs or something like that which is shared across these projects. And it might have a little bit different location here. In this example it's a common, you can see it on the left picture. Also you can have environments based approach of structures. Your projects, as you can see on the right picture with modules. When we talk about various workspaces, mostly we talk about different number of workspaces per project and different naming combinations. So some of the projects might have staging and production only, some might have life stage and so on. Also we have different structures here because some of the projects might have components which add additional level of the file structure. And in the end we might have different terraform versions across the projects and also different versions inside the project between different workspaces, because you might be in the process of upgrading terraform and you have on managing, for example, one version of terraform. And in production still you can have different version. Okay, so let's summarize our complaints. We have following pain points, different terraform versions, and it's a pain of switching terraform versions on local between different projects. Also you have there a risk of incompatibility of the snippets. And also here we have a problem that our workspaces not visible and not trackable by versions control system. And that's why you need to memorize everything. And it creates a problem when you involve someone for troubleshooting or onboard newcomers. Aren't there any solutions on the market? Let's see. So we have terraform cloud, which is famous SaaS solution. It's a native solution for terraform and it's really good, but it's a pricey solution and it's not solve everything that we need. So it still for us has a low visibility for workspaces. And you still need to memorize your structure and workspaces and the price for sure. If we take a look into other tools, terraground, terramate, terraspace, they are also really good and they cover some pain points, but not all of them. Instead they create also some pain points, because with introducing them, you need to maintain some files, they are required, or some file structures or so on. So they create some amount of overhead. Sometimes it does make sense to use it, sometimes not. But we are trying to talk about the lightweight solution. So we do not want to have overhead, almost nothing. And then three years ago I thought that. Okay, wait, I already saw it somewhere and I remember that we have Docker compose and docker and Docker compose have a configuration for the projects in yaml. It's a very clean approach with config as a code and it has high visibility and traceability. And the idea behind terracompose, exactly that. Why cannot we do something like that with the terraform projects? Okay, and let me show you this quick wrapper. So terracompose, use cases, use cases are actually here actions that we can run in terracompose, we can run plan, apply workspaces, run shell and help more often. For sure we using plan and apply. And under the hood these actions are having terraform commands and providing output of them. We also can run command inside the container with run action. We can list workspaces and we can run a shell inside the container and join it. If we take a look on collaboration diagram here, we can see that when we trigger terracompose action under the hood, Terracompose will read configuration which is written into the config file and based on that run terraform command inside the docker. Also it will run this action, wrap it into some different commands like change directory, terraform, init, select workspaces and so on. With this setup we solve our problem with terraform versions because we use Docker which allow us to have different terraform versions on the same machine. Also we solve problem with different structure and workspaces and memorizing them because we can customize our alias with different path, different workspace and we don't need to memorize them. Only one that we need to do is to actually create this config file. We also improve our validation approach because we chain terraform commands and we automatically run validation. We will not forget about it and it will also automate some cleanup things after running action. So it will also help us in this way when we will talk about capabilities. So here I would stress that we don't need to memorize it and it's a part of code base so it has same visibility. You can have merge requests and you can have a documentation just at your code base about workspaces. Also having terraform commands provide us a guaranteed validated plan and it will just speed up maintenance and development because it require less time for double check, for printing, repeated comments and in the end it means less validation failures on CI when you will push your code because it shift left validations and it require from you to do all checks locally. First in the configuration file you can see two main blocks. This is default which contains shared properties across the aliases and aliases which is actually our projects and environments. And here you can see that this terraform version that we define in default might be overwritten in alias. The required property is only one path only. This is distinguish aliases between each other and we have here also example of the hooks usage. Hooks will help us to customize our projects even more so we are able to run scripts or commands before some particular activities and or after. And this way we will be able to do some workarounds for some h cases. We will talk about it later. Here you can see the output of the help action. So it lists available actions and also it lists available aliases. And now it's time for demo. First of all I want to emphasize few points about demo. So first we will have it in AWS. Second demo project I prepare it and it contains for speed and for readability projects that creates only vpcs and security groups. And also we will keep terraform state locally for simplicity, but please never do it in production. Here I want to show how I connect to AWS. So it's just a simple AWS profile and I export also profile name with environment variable. I also created these commands, this aliases which is just help us to not print that long commands during the demo. So let's check that we have only default VPC and default security group. Okay, everything is prepared. Let's download the tool. So I'm downloading the terracompose tool and copied into the local bin. I also will clone the terracompose demo project which is available in GitHub. And let's start with the first demo project is a common so as I mentioned, it will create a vpcs for all our projects and we can use, and we can create resources in these vpcs. So here we have folder with terraform state. We have two environments, nonprod and production. And we have here tf varst files, one managing with workspace and another one does not match with his name with workspace name. So for this use case we for staging just define workspace property. And for production we define beside workspace property, we define also tfvars property. So this is a very simple example when we can omit some break in naming combinations and we can use a random name for tfwars. Just put this name into the alias config. Okay, let's build a plan. So this is just a simple command. As you can see this is plan and then alias name. Now you see the output that we are running, terraform init. We see the normal Tf init output. Also we can see that workspace Nonprod has been found in config and we will select it and we see the information about the main factors for our plan. This is workspace and this is tfwars name. Then after checking plan, you can see that this plan is actually saved in this file. So we can even manually check it if you want. Okay, then we can run workspaces alias. This is also just a short link to list the available alias workspaces. Sorry. And we see that we currently are in nonprod workspace. So let's apply. When we run apply, it's also easy. It's just apply and name workspace. You don't need to define all details about files and so on. So it's done already. Under the hood we see in which directory we are, we are running terraform in need standard output of it, selecting workspace. And now we see that, okay, we want to apply file with this name, are you sure about that? And also we can see how long time ago it was modified. So now you can decide, is that exactly file that you want to apply or not? So if we print, yes, we can proceed. And our sources were created. Now let's build a plan for production. And you remember that there we had an TfVars named differently from workspace, but terraform plan is there and file also pick it up correctly. Now let's apply. And now you see that, okay, I ran the command, but now it contains a suffix. So with the suffix debug you can enable debug mode which will run it without init, without format check and without validation. So it's very useful for development process and debugging process, because you don't need to wait until everything will be validated and in its process will finished and so on. So it's very useful for development, but I would not recommend doing that out of development. So you need to have a clear understanding for what you are doing that. Okay, so we want to apply our changes, they are applied. And also let's check how we can run other commands. So we can just trigger action, run alias and then we can put any commands that we want to run, they will be run in the container. So here you can see that we list our state and we see that our VPC is there. And also I want to show you that our vpcs were really created and they are in place. Okay, now let's check more complex example. Here we see project that have two compose, let's go with the app component. And here we see that we have beside our Tf state folder with two workspaces integration and live. We have also here some tf state for experiment. So for example you are doing some experiment in integration workspace and you have this TF state folder just for integration workspace and you want to apply that. So how can you do it for that you can use backend config property and define their workspace deer with another location for the terraform state. Or for example, if you are using remote backend, you can put here some key and use that. So that just allow you to run terraform in it with some custom backend config. And hooks. Help us here because when we will jump from one environment to another environment we will see error because we ran terraform init with different backend configs. So for work around that issue we can just run hooks which removing terraform folder on both of these environments and it will solve our problem. This is just an example, you might have different use case and let's see how it works. So I going to run a plan on integration. We see here that we trigger before df init hook and we run our terraform init using backend config. We see config sorry, init output and selecting workspace. And we can check that in the end we are in workspace integration with TFwars integration tFwars. Okay, we see the plan here, no errors. So I'm going to apply it now. After resource were created, we see that changes are actually were recorded into experiment TF state and TF state with origin was not modified. So you can see that this option really works. And our security group also in place, we can see it here. Okay, now let's try to run interactive shell into container. And for that we will use shell action. We also need to approve it because for the projects purposes and we are inside the container. Now let's check our workspace list. Okay, so now we just check in that we are in the backend configuration experimental one, and for live we can just run terra compose action plan. It will run before Tf init hook again, run normal Tfe init without experimental backend config. And you can see that here we have plan that has been built incorrect workspace with correct tfwars file. Let's apply. Okay, so the resource was created and here we also can see them. Good. I think it's enough for this demo. I hope that you saw that with this setup, terracompose tool is really helpful when we we can just run action name and alias and it will run it with the properties with the arguments that we defined in the config file and we don't have to remember them and double check validity and so on. Let's back now to our presentation. So with that said, all done now, what do you think? Let's check our points. So here we have almost everything solved switch and terraform versions. It's now only attribute in YaML thanks to the running it into Docker snippets and compatibility visualization. Workspace traceability of the workspace automate routines needs to remember all these were solved chain in commons still. Well, it's a solved, but it's hard coded. That's why I market it as partially solved, because in the future plans I want to make them customizable and also allow users to create a custom action or override default action to solve different problems, or maybe just because to fix something. And if we talk about complexity and price. It's an open source and it's a simple yaml that you can create for 1 hour. So I tried to implement it into different projects and I spent about 1 hour and the job was done. Now let's talk about the future. So since that was just a simple wrapper, a quick wrapper written three years ago. So I didn't thought about the proper interface design and this is something that I see needs to be improved. Also, I want to make actions customizable as I just mentioned. And who knows, maybe rewriting go how do you think? So I actually want to encourage you to contribute to this project. You can find links to this project in QR code and in my GitHub account. Please join. And with that I want to say thank you for your attention.