Hello and welcome. My name is Kevin Petro Givinch and I will have pleasure to talk about Azure DevOps private agents at scale. So how easy you can build your private agent pools using my approach. So go with me through the presentation and at the end of it you will obtain all the information which is needed to configure this private agents pool in your projects. Okay, before going to the presentation I would like to say a few words about manager manager senior Azure architect team, focusing mainly on the cloud migrations and greenfield projects. I really love to make automation of the solution, so it includes both code deployment and infrastructure creation. With my favorite tool terraform. I really like to also share knowledge. I do that through LinkedIn, my blog Azure way, and of course events like this. So just firstly I would like to show you two types of the Azure DevOps agents which are available so safe hosted and Microsoft hosted agents. So briefly we see the difference between them so you can easily define which agent is better for your workload. Okay, so the first thing which I would like to compare is the maintenance. So it's self hosted agents like the name suggest you need to maintain this agent by yourself. So you need to take care about the update, the networking, the connectivity, the tools which are installed, the updates of the system and so on. On the other hand, in Microsoft hosted you just use it, you do not care at all. So this is for sure easier path. Okay. The second thing, the integration. In my opinion, this is the crucial moment when you decide which agent you should use. So imagine that you have a project which is network isolated so it can be behind firewall, you can use private endpoints, you can use another isolation methods, then you need to use safe hosted agent because then you can integrate it with your network. So on the other hand, Microsoft hosted cannot be integrated with the virtual network. Of course there is also Microsoft hosted have some limits on the cpu memory and so on. So if you even do not have isolated workload, but you need more resources to build your project, then you need to go with the self hosted. Okay, what about pricing and scaling? So in the self hosted mode you have a really nice feature that if you add user with the MSDN license to your organization, you obtain one license for self hosted agent which includes private projects. If you want more license then you need to pay $15 per month. Of course this $15 per month include owning the license to running next self hosted agent. So the entire infrastructure on which you run this agent, you need to pay for that. Okay, so on the other hand, Microsoft hosted you pay $40 per month and you do not care about nothing. So the infrastructure and license is included on that price. What is really nice thing, you obtain one Microsoft hosted agent for free in Azure DevOps, but it has limit of the execution. Of course this agent for which you pay do not have such a limit. Okay, so if we are talking about private agents on the DevOps, of course you have many ways to achieve it. So this is not one solution which is perfect, but you can use many ways to achieve the goal. So now I would like you to present some of the options which we can use to create private agents. So the first will be maybe not popular one, but I really like it for certain scenarios. So this is Azure container instances which I like on that. This is extremely easy to set up, so you can set up it with only one command for the Azure ClI. So the setup is really easy, but there is some disadvantage that you cannot build docker images on it. Of course you can do some workaround. For example you can use Azure container registry private pool in which you can build your docker images and integrate with your virtual network. But as I said, this is only workaround. Of course there is another disadvantage that this agent is one instance which is constantly running. So imagine the situation that you have many pipelines and maybe you face such a situation in your projects. So when you build one pipeline and then second pipeline it sometimes fail, but in the other configuration it succeeded. So it means that your pipelines are somehow dependent on each other, but in not a state way. So the order in which run is really important because there must be certain order that these bill are failing. So it can be cases because that agent is not clear. So it has some dependent, maybe settings or some environment variables, maybe tools from the other build which was run on this agent. So yeah, we need to be careful on that and I will show you how we can avoid it. Okay, so the next thing will be of course Azure Kubernetes services, so you can create your private agent pool on the aks, which is a really great service. I think we can agree on that. And what's more, if you will use Keda for it, then you can make even driven scale of these private agents running on the Kubernetes services. So you can adjust your agents, for example to the qui build. So if you're on your qui will be for example 8910 waiting bills, then you can scale accordingly with Keda. But of course this Azure Kubernetes services is difficult to set up. This is not one line comment. So this is more difficult to set up than container instances, but of course it's much more powerful than container services. So of course with the aks you can build docker images but it requires additional configuration. And what is great thing you benefit from all aks features. Yeah so self healing, scarring and so on. So this is a really great approach. But in my opinion this is really nice when you have a big project and long running project. So if your project run long time, so this is about two years or more, then you can think about setting up Kubernetes services. Okay. And the last but not least option is Azure V two R machine's case approach. So what I like in that scenario that this is pretty easy to set up. Of course this is more complicated than Azure container instances but less complicated than Kubernetes services. So it can scale up, scale down, can build docker images and is really cost efficient because you always scale to your needs and this is pretty similar to Keda. So if you have some builds waiting then you scale accordingly with the visual machine skset, but this does not require any additional configuration. And with this last approach I would like to follow in that presentation. So I will show you how to create virtual machines cases setup with of course step to step. And for what's more I will share source code for you in which you can run your setup in your project. Okay. But of course every setup has some challenges and visual machine skset agent pool is not different one. So what are the challenges here? So the first challenge will be to build a universal agent image because if you run such an approach then you need to cover different teams and application needs. So you will have some team which use net, some which use Java, some which use node or some data teams or even some architects team which creates architecture, for example in azure or AWS or even on the other clouds. So you need to have universal image, you can call it gold image, but building such an image which is a universal one is a challenge. I think we can agree on that. So you need to install a lot of tools and these tools will be in many versions. So you need to be careful on that because you cannot update these tools with your own choice because you can break some other build. So you will end up with many versions of your golden image and you need to be sure that you update your scale agents pool in a smooth way that you are sure that some teams are not using the version, for example five and they can switch to the version six. So you need to create a new agent pool, tell the team, yeah, you need to switch to the new pool because new version we deployed and check your old pipelines, your old builds. So if they can move to new version you can decommission previous one. But still you need to version the image and I will show you what tools you need to use to achieve that goal. Okay, so how this image build process can be made. So the first important thing, and I think this is a really great information that you do not need to create this golden image because it's already created. So on the GitHub you can find the repo where the definition of this image is already done. And what's more, using those sources. I think you use it many ways but you do not realize. So these sources are used to build GitHub runners and Microsoft hosted agents. So you can build these universal agents for your own private pool. I think this is really great information. To build that image you use Packer tool. And why to use Packer? Because you can just run it. You can go install the tools and create the image. Yeah, but with Packer you have a lot of tools, you can make the test of the setup and this is done through the CI CD pipeline. I think this is a really great approach that you can automatically build your golden image and Packer do that job for you. So in the background packer create virtual machine. On that virtual machine it execute the scripts, then make some best and then create image from this virtual machine in which you can use it. And in that process you will upload the image created from Packer into Azure image gallery. In that Azure image gallery you can have many versions of the image. So you create container. For example my golden build agent and on the my golden build agent you create versions and using that versions you can create virtual machines cases it of course you can have many versions of one image, which solves our problem with versioning and different tools. I think this is a really great approach and now I would like you to show how to implement it through the demo. Firstly I would like you to present the GitHub repository. Its name is images and what we can find in this repository is the source code used to create VM images for GitHub hosted runners used for GitHub actions, and also Microsoft hosted agents used for Azure pipelines. So in that repo, as you just see is a source code which is used to build choose hosted hanas. In my case I will be using ubuntu 20 four. So now let's see what are the tools installed into this image. To see that we need to go to the images next to explore Ubuntu. And now go to the readme file. Okay so in the readme file we have all information what is deployed into this image. So we have OS version of course system d image version and so on. This also specified language and runtime which I installed package management tools. So you have a pretty long list of the tools. And of course we have also some Cli tools like Aws, Cli, azure, Cli, GitHub, Cli, Google cloud, openshift and so on. Okay so now I would like you to show how this image definition look like. So to find the image definition we of course need to go to ubuntu and now explore templates. In the templates we need to open packer definition file. Okay and in the packer definition file we find firstly required plugins. This is the same structure as for terraform. And now nextly variables. These are pretty long list of the variables and then the build image step with usage of the azure iam. Okay so this is how it looks like, but in the setup I would like you to show the installation of the tools. So you see that this is execution of the prepared scripts. And for example there is also a skip build install terraform. So now go to that script and see how it pools. So we need to go to the scripts next build and then install terraform. Okay so go ubuntu scripts build and ethernet. Okay so the installation is pretty easy as you can see we first need to download the binaries and then unzip it to the user bin director. And then is our invoke test tools terraform. So let's see how this test looks like. So again we need to go to images ubuntu scripts next best. And then tools test pds one. Now we find terraform. Okay so we have a terraform test, we have a terraform version and should gcatune zero exit code. So it's simple word. This will just return the version of the terraform. Okay nice. So now I would like you to show how looks like the terraform skip to create the packet image and the virtual machine skip. So firstly we start with our providers for telephone. So we define two. So this would be hashicop azure rm and also hashicop random. I like this library for the random because I can generate random numbers, random passwords and so on. So really nice thing. Nextly I configure backend for the terraform in the azure in that case of course I use azure storage if you do not know how to configure the backend for the terraform, you can go to my blog and see article which goes you through all the steps how to configure backend and of course how to confuse provider in the terraform for Azure. So how to create client id, client secret and how to set up subscription id and of course tenant as you configure your telephone providers then we can look into the exit terraform script. So first things you need to notice is the image path. So this is path to the Ubuntu definition in Packer. And as you can see I have the directory running images main and this is just simple copy of the repo which I just show you in the seconds. Okay, so what we need to create to build your image and to create virtual machine sky set. So firstly we need a gallery. So we create shared image gallery here. This is just our service for storing the images and the image version. Next we need to create shared image. This is just a container for your image versions. So this is not exact image but this is like a resource group for the images. So my name will be Ubuntu 2204 agent pools. Next I need to use a new heso because I would like to add packer in it. I run packet image with my image path. As you remember this is path to the Ubuntu four packer image definition. Next I run next no resource bound by but now this is a packet runner. It means to best a packer image. Of course I need to somehow authorize in Azure. So I need to again set the client id, client secret subscription and of course tenant. What you can notice in these parameters is a temp resource group name. So this is a resource group which will be used by Parker to create virtual machine and install all these tools there. And from this machine create image and this image will be placed in the managed image resource group name. Of course with the name specified in the managed image name parameter. Okay, after our skip is done I need to wait for it on the depends on. So see I must wait for the packaging to be finished before I can obtain azure image. I use that data because this is not adhesives which I create but I just made import to the terraform. And you remember that this image was created by a packet just in the packet runner step. Okay, so while I have the exact image I can create image version and put it in the image gallery. Of course I use some name in my case will be free. And of course you need to make a reference to the managed image id where in my case will be the id of the image created by Packer. Okay so the last step will be to create the skyset I use virtual machine skyset module defined in the terraform and in here I just need few parameters, resource group and location skew and image gallery image id. Okay so now see how this module look like. I create a random password. I need it for a setup. By honesty you don't need to log into these virtual machines because they are controlled by Azure DevOps. So next create just simple virtual network and subnet. Of course there is nothing special subnet needed. I think you need to adjust to your needs in your project. Next I create virtual machine skyset with the of course name admin user password for the instances I put one because this will be scaled by Azure DevOps so I don't need more. And what is really important, you need to set over provisioning to false and upgrade mode to manual as these parameters are needed to use this virtual machine sketch as an agent pool. Next I play source image id. Of course this will be id from the guy image. Okay next standard parameters, nothing fancy here. Okay so now see how this pipeline because I use Azure DevOps to build this terraform of course. So I do not want to run my local machine, I would rather use some pipeline to do that. One important thing on the pipeline is that timeout in minutes. You need to set a really high because this build cases about 2 hours which I will show you just now. Okay so we have pipeline create plan and apply. And you see that this took about 2 hours. Of course it can vary. Okay so next go to terraform apply. Of course this is some standard setup. So I install terraform, I made some in it. I made terraform validate and then terraform apply. But what I would like to show here are tests. So go and find it. Yeah, so you see that we have discovered five tests. Discovery was pretty fast. Now these are filter selected four tests to run. As you see the filter is power nodules and there is a running of the best. And you see that test passed four fight zero skipped zero, not on one. And if you find that any of the best will be fired, of course the build of the image will be fired also. So you are sure that all pools are working. In another case you will just do not build this image. Okay, so now take a look for Azure side there is an agent pool automation code resource group and what we can find it. Yeah, so we have compute gallery, we have image, we have virtual machines cases, we have some virtual network. We have image definition and image version. So now take a look how this all connects with each other. So firstly go to the image gallery. In the image gallery you will find definition. So this is our Ubuntu 2004 agent pool. So you see that this is only image definition which I can rely when I try to create virtual machines case it. So go to this definition. Okay, so you see that I have some versions and using that image definition I can create virtual machine or I can create virtual machine skyset. If you go to the version you also have create VM, a create virtual machines case set. But then when you use it you will use the exact version of the image. So in my case will be free. What is really nice is the update replication. So you see that we have a target region, replica count and stripe skew and replication status. So when you firstly run out form, the replication status won't be completed, but will be in progress and you will see a percentage how much of this application was completed. Of course this replication status must be to completed. Two this step on the telephone will be finished. So as you can see you can put this image in many regions and also use many versions. So if you would like to build another Ubuntu 2205 ZFO agent pool, you will just put another version to that image definition. And then next you can use it with the virtual machine skill set creation. Okay, so now we can go to find how we can see how we can create a private agent pools. So we need to go to the organization settings. Next go to agent pools and in the next you need to go and click add pull as a pools type we'll choose virtual machine skset project for service connection. In my case will be azure way Azure subscription. So our connection will be enterprise subscription. Next I created a special virtual machine skate set for this demo. We specify the name, okay, and now we have few options to choose. First is an automaticality down virtual machine after use what it means. It means that when your build finished then your machine will be destroyed and new machine will be created to use for the next build. This is a really nice feature because you are sure that your agents is clean. I do not have any dependencies from other build, so for sure we want to turn on this feature. Next, the maximum number of each of our machines is Skyset. I think the description is pretty obvious, but you cannot set more than you have parallel jobs. So in Macari this is a nine, okay, so numbers of agents to keep on standby. So this will be the number of the petrol machines which are waiting for the bills. So if you want to save money, just put a zero here. But you must agree with that, that for the first build you need to wait a little bit. This will be about two, three minutes. Just to this agents will be connected to the delts. The last parameter is delay in minutes before deleting excess idle agents. So it means that this is a time, how much time this agents will keep on standby. If there is no action in that 30 minutes, this will be ticked down. So if you have really big number of the bills waiting, then this number won't be used because these agents will be occupied and you will never touch this delay. But if for example your team stopped working at this day and because this is an end of work for now, then probably all agents will be teamed down with this delay. Okay, so of course how to provision this agent pull in our project. It depends on your needs. Okay, so if you don't want to have this pool be available on all projects, just turn it off. Okay, so when you click create, then this pools will be ready in about 15 minutes. So you must be patient here. 15 minutes is beacons small time. So go to the, now we go to the agent pools. Again, there is an example, this is example pools as you can see on the agents. This is built on the example virtual machines kset. And this is also made on the same image which I just showed you. So I would like to go and present you that using this pool I can build the docker image. I will just make a run pipeline here and show you. Yeah, so you see that I have a really simple pipeline built and push state and there is a docker build, easy one, nothing fancy here. Okay, so I just run this pipeline, it should be finished in about three minutes, maybe two. So we'll back to it in a second. And as you can see, one more thing, it uses the example bitchuma machine, skset agent and of course the machine, the name is the same. Okay, so we back in a minute to see how it goes. Okay, so the summary of the process, it looks like that. So first we need to create shared image gallery just to have possibility to later on use image to create virtual machines case it. And next we create shared image. Remember that this is just a container for the image versions. Then we unpack an image build. To build our image as an output we need to create image version. So after we create the image version, we can create virtual machines cases. And the last step will be configure Azure DevOps agent pool okay so maybe we see how our build is going, we need a few moments more. Okay so what if of course I had different issues when I tried to build this golden image. So the first was we stopped hearing from agent, from the hosted agent okay so I tried to use Microsoft hosted agent and after one and a half hour I got that this hosted agent was disconnected. If you will put this error into the Google you will see a lot of issues with the hosted Microsoft agents and of course nothing was resolved. So in order to build that image I used another private agent. I set it up on the virtual machine, the Linux one of course the Ubuntu so then it all worked just fine. But I have another arrow. This was fate to fetch four fee for Biden. So I got this arrow, why don't unlink one of the pools? Maybe that was, you know, some issue with that server. I know how to say when I rerun this process the image was built without errors. Okay so yeah you see that the build is already finished. It took almost two minutes. Let's see how the agents looks like. So you see that my agent is offline because I have a settings that it needs to be teared down after the build. When I will another pipeline then I will go to fetch agent. Okay so what about the costs? So in my project I run it for one month, I set it up maximum number of agents to 15 and the build was $150. So it means that I paid $10 pay per agent. Of course this $10 is only the cost of the infrastructure. So this is cost of virtual machines cases. I think this is a pretty good price as the hosted agents, the Microsoft hosted agent it cost about $40 per month. So I use the same image as the Microsoft hosted pay $10. So I think this is a pretty good and of course this cost may vary because you can set up another number of the Eden agents, you can have more bills. So it all depends on the project but you see the overall cost. Okay so if you would like to obtain the source code and description for the terraform which I just show you and of course the pipelines, you can go to my blog. This is Azureway cloud, you can scan this QR code and see the description and all the needed URLs. So the GitHub, GitHub runners and the GitHub repository of my Azure way blog. Okay if you would like to stay in touch with me you can just follow me on my LinkedIn and if you have any questions of problems with my examples you can contact me on the private message. For sure. I will help you with that. Okay. So thank you for your attention. I hope you enjoyed the session and I hope that you will use this approach in your projects. And the setup will be really smooth using my tutorial and source code. Bye. Have a nice day.