Hey everybody, and welcome to this session on maximizing CI CD efficiency, where we're going to dive into reusable templates, specifically with azure pipelines, and how to make the best out of it. And love this quote from David Farley is that releasing software is too often an art. This idea of an art is that every picture or every painting you do is different, when in actuality it should be more of an engineering discipline. We should be able to crank these things out very similarly and get all the best constraints with them in terms of how we want to update them and maintain them over. And that's what we want to dive into today. Before we get started, my name is Travis and I work as a distinguished engineer at SPS Commerce and you probably haven't heard of SPS commerce. We're a business to business organization that focuses on connecting the world's largest retail network together, which is retailers and suppliers to exchange purchase orders and invoices with some of the biggest names that you might expect, including Walmart, Target, academy, sports and Bass Pro, and many more. My focus at SPS commerce is specifically on developer experience, and you might be asking yourself a little bit, what do you mean by developer experience? What exactly is that? That can mean so many different things. For example, if I change my address with the HR department and I'm a developer, is that a developer experience? And the answer is a little more fine grained than that. And the definition that I like to land on is that developer experience is the activity of studying, improving and optimizing how developers get their work done. We're focusing on how you get that production, sorry, that value to production and everything in between that affects your development process and workflow. And so we're really marrying together this idea of user experience along with your development principles, specifically at your organization to form your developer experience. Your experience might be specifically shaped by the tool set that you have, by the process and the workflows that you have, and by the best practices that you have. In a lot of cases, our organizations have been around for 510 1525 plus years, and that means that a lot of these organizations have a series of tools that just have popped up or grown at different points in time for different initiatives. And that's why I think that this problem statement really helps define what we're trying to solve, and that's that developers work in rainforests, not planned gardens. We didn't necessarily have an opportunity to plan out the workflow of how you deliver what you're pushing to production. Instead, you were given this tool and you're told to do this change management process. And so that leaves you as a lead developer and a lot of times fending for yourself and figuring out what that workflow is. And it might not be the best since you're spending your day on delivering value to production, not actually delivering an update on how you get that value to production, the process in between. So at SPS commerce we focus on developer experience capabilities, and we think of these capabilities as identified horizontal fast tracks that we want to curate for maximum productivity across the organization. What are the things that we do most often and how can we make them better across it all? And so if we draw the organization in these common sets of verticals, we can begin to think about these types of stories. Like API design is a really good one for SPS commerce. We are focusing and zoning in specifically on our API design process as an API first organization and how we deliver great APIs. You can also think about code, reusable and inner source. And how you share code effectively across the organization can have a large impact building deploying a new application from scratch is sometimes fraught with so many different toily type of things where you're trying to figure out how to build inside your ecosystem. Or maybe it takes just way too long to figure out what those best practices are for building a new app. And of course, what we do every single day as developers and engineers is to build out and deploy new features to production. Think about your process for approaching that and all the tooling that goes with it, and your change management and your feature flagging and feature toggles is all wrapped into that. But what we find across all of these as we think about our developer experience, is that CI CD is at the core of it. This is our orchestration engine for enabling a lot of the value that we want to keep throwing, sorry, keep moving through these pipelines, right? And so that's essential to land on a piece of common tooling that can help you do that and help you land on specific patterns that make that easier. Whether it be reducing toilet or keeping developers in their creative flow, or just reducing duplication, having a standardized CI CD practice with effective methods for rolling out new pipelines can be essential foundation in order to do that. And Azure DevOps is one tool that can do that pretty effectively, I think. And if you're not familiar with Azure DevOps, it's a series of products and services that come together as a SaaS offering. It does offer work items as well, which is similar to like Jira items. It does offer Azure repos again, much like git repos, it offers Azure pipelines, and many of the features you'll see here is very pluggable. And so actually my experience is mostly with Azure pipelines, specifically not with repos or boards or test plans or artifacts, because you can plug and play different components that you want to use out of the Azure DevOps ecosystem. And pipelines is an excellent example where it works great with GitHub repositories. It works great with connecting to Jenkins as a source or a destination. There's lots of different opportunities that you can use it in these tight scenarios or situations. Used to be called, or I should say it came from team foundation server or TFS if you're familiar with the history of that particular product. Eventually it became visual studio team services and was available online, also known as vsts, until the name that we know today as Azure DevOps. It is a SaaS offering, as I mentioned, and that means that it is available in the cloud, but you can have self hosted agents as well as cloud hosted agents, much like you see in GitHub actions and other systems. There are agents that are ready to go to do your bidding if you want to pay per hour per minute for those, but you can also deploy your own agents, which can be very effective way to integrate with your ecosystem with your security. Best practices. From a cost perspective, I think you'll actually find it's very cheap to execute if you're already in the Microsoft ecosystem. It can honestly be almost free depending on the size of your team, depending on the type of license, and depending on what you want to use. Azure pipeline specifically actually only requires a stakeholder license to use most of the features that are there. And so that's something to consider that this is a very cheap offering if you're already a Microsoft ecosystem, and if you're not, it's not even that cheap to get. I should say it's even cheap to get started in that way. They do offer a marketplace where you can download and install custom extensions and tasks, though they can be a little fraught with some staleness depending on the ones that are there. You want to do your due diligence before entertaining the installation of some of those, and it's important to remember that when we talk about Azure DevOps, even though it's got that word Azure in the name, it's actually cloud agnostic. Sure, they provide integrations specifically for Azure that are there by default, but there are also extensions you can install specifically for GCP or for AWS and it works great into them. In fact, we'll talk a little bit later about our example and current architecture, but we deploy actually 80% of our services into AWS and do it entirely through Azure DevOps. And it works great as a platform orchestrator for that. And so as we position Azure pipelines, then within side the industry, I think you'll notice that it's very well positioned. If we look at the g two quadrant, it's definitely there towards the top as one of the leaders along with GitHub Actions and GitLab and several others. If we look at the postman report from 2023, I think you'll find that Azure Pipelines is number four on that list. We see that top four being very common, GitHub, Jenkins, GitLab and Azure pipelines. Of course GitHub actions is fantastic for open source, but sometimes you really need those enterprise level features which you can only find in enterprise level software like Azure DevOps and Azure pipelines. And so you'll find definitely some additional capabilities, especially in some of the templating language we talk about today that is not present in some of these other tooling that I think is a large advantage. For example, when we think about GitHub workflows, they're great, they provide a lot of capability. But I think you'll find the templating features and some of the things we're about to accomplish today are a little bit easier, a little more straightforward in Azure pipelines. As we look at Azure pipelines, again, if you're not familiar with it or you're relatively new to it, we really want to focus in on producing what we call the YAML based pipeline. And focusing in on YAML is essential because YaML allows us to define this definition of our pipeline as code. And if we're really going to take advantage of some of the components of Azure DevOps and reusability, we're going to want to do that as YAmL syntax in our code base. Here's an example pipeline, and this is actually a real deployment pipelines in my organization. And here you can see we have lots of different executions and things that are happening. But it looks pretty straightforward in terms of the workflow here, right. This is the YAMl workflow where we have a context analysis. We have a compilation stage before an integration, before approval, and finally before production deploy with parallel streams of deployment to multi region app that's going out behind. And so in here we also have other components that we might want to look at, tasks and checks we won't have time to cover those today because we're focusing on these templates. But templates will give you that foundational basis for how you're going to integrate and work with tasks and checks and all these components together. So let's dive in. Let's talk about templates and this idea of building a YAML templates. YAML templates in general are not something you should avoid if you're worried about writing too much YAmL or becoming a YAML engineer. I'd say that is partially the world that we live in currently. But to a certain degree this just enables so many other capabilities. There are so many advantages in doing so that it deems necessary. I think you'll find that on the tech radar for thoughtworks, which release information about when you should trial something, when you should adopt it. You'll find that both Azure DevOps pipelines as well as YAMl show up here and are present on something that you need to be trialing and trialing, not from the sense of it may be or may not be ready, it's ready. It's just something that you need to look forward to using and see how it fits in your organization. So there's a lot of traction here. YAML templates is a pretty standardized approach, and today, as we look at templates, templates are defined as reusable content, logic and parameters. Templates function in two ways. You can insert reusable content with a template, which is great. That's kind of what we expect, right? Just take content, put it in the spot and materialize it that way and render it. But here's the interesting part that we'll talk about a little bit later is that you can use a template to control what is allowed in a pipeline. So there's a security aspect to the templates that we'll see here as well. So we'll talk a little bit about the anatomy. What are tasks, jobs, stages, pipelines, how do they all fit together? Before we dive into the ideas around extending and including, which is basically inheritance and composition, and if you're coming from the object oriented world, you know that those are great things. We just have to figure out how to use them appropriately in the template world. So let's get started. Let's think about our functional requirements, and our functional requirements here as an example. Typically for a build, well, what are the things we do? We have a shared context or variables or configuration across the organization. We have some types of security constraints or deployment environment or build environment. We have versioning that we need to do, which could be semantic versioning, could be infrastructure style versioning with the date as a part of the version name and an iteration number. We usually have private packages that we need to worry about coming from, like an internal JFrog or GitHub artifacts or Azure DevOps artifacts. We also have build verification. We got to unit test it. We're going to push artifacts, maybe a docker container, maybe some other files and artifacts that we need. We also want to tag the repositories, let them know that here's the immutable version I created that's associated with this. We might have some notifications on failure or success, and all these are fantastic, but boy, there's a lot of things I have to do and consider every time I build a new pipelines, when in fact a lot of these, other than these two, these other ones are really the same across most of the pipelines that I need to build. And so when we think about this content that is unique to each pipeline, the differentiation is really how do I build this thing, how do I verify it, how do I validate it and then unit test it? Pushing artifacts is arguably the same across it all, and I think we'd find that also true for deployment. In deployment we can look at organization, we can look at security constraints, change management, artifact promotion, and a lot of these are the same across every pipelines. And so it feels awkward to copy and paste that yaml every single time. We want to get away from that. We want to enable ourselves to be able to make changes in one spot when we want to update and have less code in our repo. And that's where the templates play an important role. Even here you can see that depending on how declarative your logic is, there's just a lot of boilerplate, and you might not even have to have any deploy specific logic other than tell me my environment and I should know how to declaratively deploy from there. So there's a lot of boilerplate. This idea of code reuse is really, really important across your organization, especially not just in your pipelines, but also in your code distribution as well. I have another session you can find from other conferences called compelling code reuse in the enterprise that I think you'd find interesting. If you're enjoying this topic. So let's dive into the pipeline anatomy then at a high level. This is the diagram we're given from Microsoft on Azure, DevOps and how it works. We have a trigger. A trigger is like a commit to a repository, and that trigger could be from GitHub could be from Azure. Repos triggers a pipeline and a pipeline is made up of a series of stages, and stages themselves are made up of a series of jobs. And jobs run on a particular machine or agent in some location. And each job then executes a series of steps or tasks that are associated with them. And those tasks can do anything. They could be script tasks. And here's an example then demonstrating that we saw this a little bit earlier, of the different components that were part of my Yaml pipelines, you can see that these are stages, these are jobs. And then here you can see are some checks. Checks are kind of run at the job level, but are aggregated to the stage level just to visualize all the checks that those jobs have. But there's also different types of jobs that we need to be concerned with. There's two types. The first type of job is a regular job and it's literally just called job. And the second type is called a deployment job. And a deployment job is a special type of job that has additional constraints that we can connect to an environment for deployment. And it's essential to use those deployment jobs for deploys and then use other types of jobs for other utility like building or other context evaluation. So here's what a YAmL file looks like. That's very very minimum. We have a trigger which is main, we have a series of stages, then in this case two stages, a build stage and a deploy stage. Jobs inside those stages. And so here you can see, here's a deploy job looks a little different. It's defined as deployment versus just job that you can see above. And in this case it's important to recognize some difference in behaviors. First of all, stages actually run automatically in sequence from each other. So we know that the build stage and all of its jobs are going to finish before the next stage is going to run, unless we decide to change that order. You can adapt that. So stages run by default in sequence unless you specify otherwise. But jobs, jobs always materialize in a stage. They by default run in parallel unless you change that. So that's an important difference to make and understand in how you can execute your workflows and build templates. Let's dive into some template basics. So first is, how do we define a template? Well, normally in a pipeline, assuming you've built an Azure DevOps Yaml pipeline before, you would define an Azure pipelines YaMl file, much to like we just looked at, but then to define a template, well, we're going to assign some parameters at the top the assignment of parameters essentially makes it a template, though it doesn't mean you could still use parameters in a root level pipeline. It's just a nomenclature and a terminology. So here we can define some primitive parameters, in this case like a name, and the name could have a type. It's usually the type that you want it to be. It's a primitive type, so it could be string number, boolean at a high level. You can also define whether it's required or not by passing in whether it has a default. If it doesn't have a default, then it's a required parameter, which is pretty straightforward. You can also enforce restricted enumeration. So we can say the type is a string, but we want to ensure that it is either value of dev or prod for our environment. And we can also define a series of stages that we might want to include, or these are stages that we're going to inject with, that include or that composition into wherever we use the templates. And so in this example we're injecting a stage and that stage uses some functions and some variables. And so here we can reference and see how we can use parameter variable syntax to reference and change the stage name based on the environment. We can also use these functions to alter or modify the parameters that we might want to apply. So in this case I like having my display of my environment always in uppercase, a nice uppercase prod or uppercase dev. So we can do upper in order to transform that. And you have many other functions that you want to get familiar with, whether it be equality checking or greater than length. You can use or operations to combine and join these together. Starts with you can invert it and use a knot as well. And then you can see here where we're just injecting a bash execution script that just says echo. Hello and then gives the name of the person that we passed in, in this case hello Travis. Pretty straightforward template that we can make use of. Now in another azure pipeline file template expressions allow us to go one step further with how we think about our parameters. So here, recognize that we can have complex types, so our parameters can actually be of type object and that object now can be a list. So notice that the default, not the restrictions, but the default here is saying I'm going to pass you dev and prod. Here's all the environments I want to deploy to. We can also change that around a bit and use complex environments where we take a list of complex objects. And notice this list below has some metadata about each environment. Now approval required, the approvers that can approve that environment, that sort of information. So when we create our deploy steps, oh, before I say that, I should say you can also have a special type of parameter here that is a step list and that allows you to actually pass in steps so we can have a user of the template actually provide you. Here's steps that I want to use and it recognize steps as a complex type. Same with job and job list, and same with stage and stage list and deployment deployment list. So you can pass in these as well known types. When you define then your actual stages you want to inject, we can begin to do some interesting things from a template expression. First of all, we can just straight up take the deploy steps that they passed in and include them. So if this is a step list, it's going to automatically include all the steps now in that particular part. So we can begin to piece this together and put them in the right spot or surround or encapsulate other tasks that are executing. But we can also do loops. Loops are pretty cool because here I can take our environment list and for each environment now I can echo out a particular statement. But think about how we could use this if we had complex environments as well. We can loop through those and here we can loop through them and we can check to see if approval is required in that environment. And if approval is required, we can take a different action than if it's not. In this case, I'm just doing an echo statement that indicates that. But you can see how you could swap out your behavior inside these expressions to build out multiple steps or jobs. You could even have different stages that are being looped through and added automatically based on that, which is when we talk about templates. We also have variable templates, and variable templates don't have parameters at the top, they're simply a file and they only define variables and they look like this. Here's an example of where I might have an environment name and a base URL, and this is my dev variables. I'm using the name dev variables Yaml very explicitly here because I'm going to have a duplicate of that and I'm going to call it prod variables. And here I can have my base URL and my environment name. And now if I were to create a template and this template, let's say was for deployment, I could use those variables contextually. And so when we come in here we can import one of those files. And so this is the same way that we'll see in a minute that we can import regular parameters and, sorry, regular templates here. We can also dynamically now bring in a parameter for our environment and import a file dynamically based on that. And so we can automatically pull in either the dev variables or the prod variables and then begin to reference them appropriately. Notice the different variable syntax that I've used here as well. Parameters environment is obviously coming from a parameter above versus the dollar sign syntax with the round brackets you see on the bottom for environment name. That's coming much later, that's coming not at render time, but at runtime. And so it's important to understand the different variable syntax that is available to us when you're completing not just a pipeline, but specifically when you're working on templates, because those are happening at completely different points of the lifecycle. So here you might see that I have macro syntax. Macro syntax is at task execution. It happens basically at the last possible minute and is like checking an environment variable. It's perfect for pipeline task configuration as the typical variable that you would use when building out a regular pipelines. But the template expression is a little different, and it is obviously used at compile time or that render time when we're rendering the template right. And it's important to note that we can actually change values of the left or the right side using this, meaning we can actually change the key value in our template, not just the value of it. And we'll see why that's important later on in a little. And of course this is necessary inside templates. Your runtime expression syntax happens after the templates has been compiled, but before it's actually running a job. So it's happening kind of at runtime, but not at the last possible minute. It's kind of in between. And this is important to use for certain type of conditional definitions where you might decide that this stage only runs in this particular scenario. And so as we have those three types of syntax for variables, you use them in different ways at different times, but this is the order. Template expression renders first, macro syntax last, and runtime right in the middle, so you want to be familiar with those and when to use them. The template expression syntax and the macro syntax are pretty standard on when you would use them, but it can be a little bit confusing on when you need that runtime expression syntax. It might take a little bit of experimenting depending on where you're using it. So when you're ready to include a template locally, then like we just saw with the variables template, you can do that pretty easily. Here's an example where we can pull in our particular stages and include a template. And this template we're including here, notice is in line with a job. And so this is a job template because we're adding it right at the right indentation level where we would add additional jobs. And notice that we can include multiple templates within your particular file so you can build up and use composition to add many of these together, and you can use the same templates multiple times. Of course, that's where we get another type of reuse in there as well. We know that deploying is pretty standard. Whether I'm going to the dev or prod, I just need to change some of those variables and that's what you can augment with the parameter. Notice here where these files might exist, there's folders. And so if we look at the structure of a repository, you'll find that if I had an azure pipelines yaml file at the root of my repo, I might have a deploy folder, and then inside there I would have these different templates for stage or build prep or my variables. Remember if we were including variables as a part of these too. I've also been very intentional with the naming of these, and I would encourage you to think about naming of your templates as well, because whether it's something that provides job level composition and include, or whether it's step level or variable, I like to name them and actually include a suffix in the name that says job or stage or variable to really define exactly the intention of that. When you're including them, it's always relative to the file that you're including it from. So as you walk through that chain, it does become easy. It's not like you're always referencing it to the original file or to the base file. Templates in parameters to other templates is tough. That means that if you're taking in a parameter, and in that parameter you reference another template, which one do you refer to? It's actually the original template in that case, and that can be very confusing. So you want to consider that maximum of 20 levels of template nesting and maximum 100 separate Yaml files total. These are pretty big limits and they want a maximum ten megabytes of memory parsing before it's going to blow up on you. I've done some fairly complex templates and haven't reached that particular problem just yet, so I think you'll be fine to build something fairly complex here if you wanted. But of course the power comes in when we're not talking about including just local templates, but the idea that I want to include a template remotely and remote is important because it means that I can make use of it across multiple repositories without moving it around or doing something special to make that happen. And so to include a remote file we do that first by adding this resources section at the top where we can include other repository information. So here for example, we're going to include our repositories and its name is Azure pipeline templates. You might create that in your repo and it is going to be named templates. And in its directory, in its structure, it would have our deployed template at the root along with environments specified underneath it. And then you can see here where we start to use some of the resources. So we're going to use those resources further down and inside. Then this line here where it says templates and then it says at templates, meaning that it's actually referring back to pull all the templates from that particular resource. And so this is a special syntax with the naming where it connects that repositories name to it. You can see later on where we actually use the term at self. At self is a specialized term that you can always use that indicates this is coming from my primary repo that is connected to this pipeline. And so that's a local file versus a remote file of what we're building right there. And this is pretty important, this is pretty powerful, right? This allows us to have a centralized azure pipeline template for our variables, for our workflow that can now impact all organizational pipelines. That in and of itself has a little bit of amazing power and also concern. And of course we expect that at the same time. But this idea that yeah, in one line change, in one character change, you could actually blow up every pipeline in your organization if you have some very highly reusable capabilities there. So that being said, you want to make sure that whatever repositories is hosting these shared templates is going to be locked down, secured and require some approval process before engineers and anyone is allowed to submit there. Ideally some good validation and pr status checks as well. It's also worth noting how you version. So in this shared repository that you might create to host your templates, how do you want to version those? In this case it's just referencing whatever's in the main branch. But there are different organizational template versioning strategies you might want to consider. The first is branch based. You could actually just have different branches where you have a v, one branch and a v two branch and you could put those templates in there and then your v one branch has everything for v one. But that couples our template versions together. Meaning if I had a build template and a deploy template, they both have to be version one inside there. And so if I intend to make a version two of one that couples a version of the other one, so I have to make a version for it too. We have non immutable usage, meaning that I could come in and make an update to that template and roll that out to everyone very easily. I think that's an advantage. I don't think we want immutable usage here. And of course it's an explicit reference that I include with that ref command that's there. Now this is similar but slightly different. You just go with commit or tag based, especially if you go with commit based, you can reference the actual commit. This is again still coupled to the same version of template that's there, but it's immutable. I can guarantee if I'm worried about it, that no one's going to change this workflow on me under the covers. Now, maybe that's good in certain situations. Again, I don't think that's a benefit in the organization where we actually want the ability to change those under the hood and maintain it, but it's worth considering. And of course we have an explicit reference there specified with the commit. Or if you are using tags, you can obviously move tags around. And so that's another option as well. But my preference is the naming option, which is what I've been kind of producing so far. This idea of a version actually in the name. And this has some benefits. It's kind of like API version. I'm just going to use the major version, I'm just going to call it v one. It's decoupled template versions. So in my main branch I have v one there. But that might also sit beside a v two and a v three. And that might mean that my build templates, that is version at v one, might just have a v one. It doesn't need to have a v two, simply because this template does. So we've decoupled those versions across templates. It's non immutable. So I can make changes to that version one and maintain it appropriately under the covers for its implementation. And notice that there's no explicit reference here. This is kind of nice. My teams don't have to worry about exactly which version do I use. They're just not making any explicit reference at all. And that's going to automatically grab the default branch and allow them to use those templates that are out of there. I think overall this is a little bit simpler in terms of git maintenance as well. You can follow a trunk based delivery pattern, which is nice, that's great. We see how to compositionally include templates, but how do we extend a template and what are some of the differences and why we'd extend it as opposed to include it. So extending a template works with the key command. That's the extend option. Extend basically allows you to provide one template with a series of parameters that you want to use for that particular pipeline. You can only inherit one templates, that template itself can inherit other things, but only this template can be used inside this particular pipeline. So when we extend it, what might we be doing with it? Well, if our base template looks like this, we might be including a stage and a stage list. And maybe before someone runs that stage list, I always want to run another job. Well, you can see that we can do that here. So we include a stage list that we want to encapsulate logic around. We can eliminate boilerplate and automatically select like an agent pool for deployment if we wanted, or for build. We can put that in so that the team doesn't have to worry about which agent they're using. We can add standard variables as well so that those are just part of the context now that I can use in my pipeline. And we can of course add that kind of context job and do whatever custom logic we might have there that we might want to run across the organization, which is pretty, so this is pretty powerful, right? But I mentioned earlier that extension is going to help us with security and some security aspects and limiting what teams can do. And of course we can do that a little bit through parameters, but there's more powerful capability if we go into advanced templating that we. So if we take a look at our base template, let's say it looks like this, similar to what we just saw when we're ready to include the dynamic stages. Here's a wallet text, but just follow me through this a little bit. It's complex, but this is going to allow us to do some interesting capabilities. First of all, we can loop through all the stages that are coming in, and rather than just one line to add all stages and all content, we can loop through that and begin to make decisions about which portions of the stage the user gave us that we want to add in. We can disallow certain syntax, like we could restrict deployment jobs or we could only allow deployment jobs if we wanted. We can apply defaults and say here's some jobs that materialize not just before but after or in between some of the configuration that's there. And if we had configuration like, do you want me to always download artifacts? We can actually include that for you as a part of the first step on every job that you add. So that way you never have to worry about pre downloading artifacts or doing certain behavior to authorize for JFrog or for other internal private repositories. Here you could just say, hey, I'm going to do that for you automatically and inject that step at the beginning of each job, even though you gave me a stage list, which is incredibly powerful if you can walk through this syntax. So here we can say download required, you always have to have these defaults. Maybe I could even run a required bash script at the start of every job that's going to bootstrap or do something. But I can also apply restrictions here. If I never want you to be able to run command line or bash scripts, I can actually remove those. Here we can add an if expression and check to see that the step name is a particular task and if it is, I can error out. That's pretty cool. That is immensely powerful, but these are really hard to validate sometimes. So when you're doing this, make sure you have jobs that help you and automation that helps you do that. And recognize that from a security perspective, this allows you to work with the security team, work with certain tasks you don't want people to be able to execute and ensure that when they use your base templates, you can never do these particular behaviors, or you always do these particular behaviors in every job. So there's some immense power there that you can begin to codify your best practices in your organization right into your CI CD pipeline. And so with these benefits then of inheritance and composition, think about some of the things that we're gaining out of this, right. Here's an example of a template that would, or I should say a pipeline that uses templates to do some really powerful behaviors. So we have a base templates that's getting us some security and best practices and default variables and everything into our ecosystem. We then have a build stage where we're including context and versioning that was just standard. We might have a Java application. So we pull in the java specific template of how we standardize our approach. We're telling it the docker file and the build command. So that way I don't have to worry about that. I can still apply customizations through post build steps if I want to codify that as a parameter, and then I can deploy very declaratively because I have all this pre baked and ready to go and use it multiple times to deploy either to dev or to prod using the exact same logic that I can guarantee since it's using the same template. So when we talk about these ideas around appropriate coupling, appropriate coupling, remember the dimensions of the architecture that should be coupled together to provide maximum benefit with minimal overhead and cost. And that's exactly what we've done here is applied the principles of appropriate coupling to create something that allows us to maintain it easier over time and provide value to the consumers of it. And this is all about removing undifferentiated engineering. A lot of this stuff. Why do teams have to continually reinvent the wheel or rebuild how I version or how I are going to authenticate to private repositories or build my app, which is always built the same way, and in the name of platform engineering, which we're all kind of moving towards these days, and internal developer platforms, what a great way to build and compose and maintain templates for how you deploy effectively to your internal developer. And so I think this is a great developer experience that offers fantastic productivity. And this is all about the ideas of evolutionary architecture as well. If you're not familiar with that term. It's an evolutionary architecture supports guided incremental change across multiple dimensions. And so this is about guiding incrementally our teams being able to add functionality over time and also maintain and help them worry about the things they need to work with, which is getting their value to production, not all these little tidbits about how to do so. So with that in mind, here's an example architecture. I'll show you how we do this a little bit at SPS commerce based on what we've learned so far. And so here's a state of the union or a state of overview for SPS commerce and how we think about Azure DevOps. We have 60 plus projects, we have 40 plus teams now, closer probably now to 500 users in the last little bit. We run 500 daily pipelines, 15,000 monthly deploys, and we run over 1300 monthly production deploys as well. We are running the classic ecosystem, some definitions, but most importantly we're using these templates to roll out over 2000 pipeline definitions today that our teams are participating in. And so our design goals and requirements and constraints as we got into this were much as we've talked about we wanted to eliminate team management and infrastructure, eliminate redundant CI CD feature implementation that we saw teams doing. We wanted a single tool for building and deploying where we could compose a pipeline in Yaml that went from build and everything. I need to drive context all the way through to declaratively deploying the components. We were tired of click Ops, which classic releases was great, but there's way too much clicking around in there. And of course we needed something as flexible for polygon ecosystem, so we could build different templates for Java or for. Net or for python and different web frameworks in there. And we wanted to build custom workflows and orchestration and governance. We knew we wanted to adapt these pipelines and build in change management, automation and other capabilities around our deployment to make it easy and simple for our teams. Of course, most important that we just talked about, it has to be evolvable and incremental, right? We didn't start from nothing and then build everything. We started very slowly and built some very simple templates that provided us what we needed and built on it from there. And we're able to provide additional functionality to teams over time very easily. So our template composition looks a lot like this. Where we have a base template, we version that, we have a context job that establishes who's asking for this build. How do I cost this build? What cpu is it related to? What are some of the variables I might want to gather as a part of this to make available? And so that allows our teams to very easily access slack channels and a bunch of other information as a part of their build that isn't normally part of Azure DevOps that we've now added. That is established through some root variables that are there. Code scanning in our case this is GitHub advanced security code scanning stages can be easily added and they can be in a couple of lines of syntax. You can have your application being scanned. For security perspective, we have different build stages for different components, especially a UI standardization where we build our UI applications and static assets and cdns in a very specific way that teams can just drop in and use with a couple of lines. And we have a fairly complex deploy stage and templates inside of templates here. So the base templates is a requirement to use the rest of these because the rest of these templates rely on variables that are established in that base template. And similarly we have a deploy workflow stage, which is a general deployment workflow and it has specific environment variables for the environment you're deploying to. And then we have a specific deploy job and it has a deploy workflow job. And there's, as you can see, a fairly complex tree structure that you can walk through in order to develop the reusability of your pipelines, but it can be very helpful to do so. So you're not having that logic copied, even in multiple spots within your repository for templates. A lot of these are not available to teams to use. These are like internal templates and really they're just using the top level templates. So the base templates and the deploy stage and the build stages, the rest is kind of built and included as a part of those automatically for you. We also have a documentation deploy stage, which again makes use of the deploy workflow in order to add documentation tasks. This makes it really easy for teams to get up and start running. And our pipelines look much like this today, where they can just compositionally add in the right components and build that together for what they're doing. Some considerations though, like I mentioned, we didn't start this from day one. We didn't build up that level of complexity on those templates. You don't want to start with the great pyramid. You can start with some compositional templates that add functionality, and I think you'll find that engineers will want to engage once you get a little bit of momentum and seeing like, oh, you mean I can just drop this in and it knows how to handle this functionality or this feature of the pipeline? Yeah, let me start including those. You'll establish a base for how you can start putting together that base templates and rolling out a bit broader. Some questions we're still figuring out, how far do you abstract some of these templates? Is abstracting too much knowledge a bad thing? What about retirement and deprecation of the templates and how to think about that like we do with major API versions. And of course there's lots of components inside Azure DevOps that you can take advantage of, including deployment jobs and the deployment job strategies that it presents you. And so it'll be really interesting to see how you take advantage of some of these templating capabilities. But one of my favorite things to look at is the law of diminishing returns in this type of situation. If you're not familiar with it, it's a principle stating that profits or benefits gained from something will represent a proportionally smaller gain as more money or energy is invested in. And I think that's very true with duplicating pipelines where once we start copying those and then you have a small change to make to one and you have 100 of those resources and you have to make changes to them all. It's really, really frustrating. So you need to look at abstracting that reusable content to something into a reusable pipeline. And of course reusable pipelines we see as the opposite. We see an initial cost, yes, is the same, and to even make it reusable one time is this high cost that might even skyrocket higher than what's shown here. But of course, over time we see that paying many, many dividends as there's updates and changes to make, it cost me less and less. And so thanks for chatting with me today about CI CD efficiency and reusable templates. If there's any takeaway you have from today, it's that the progression towards yaML templates and workflows that you can reuse not just within your team but across your organization are pivotal, especially as we start to connect ideas around platform engineering and some of that productivity capabilities. And I like this quote as well, which talks about the idea of software invisibility. When done well, software is invisible and your templates can be invisible to your teams that need to use them and that's when they're best positioned. And so when we talk about invisibility, we think about templates invisibility technique of azure pipelines. We didn't have time to talk about environments, approvals and checks and custom tasks. These are other techniques that you can use as a part of your templates in order to customize it to an even further degree. And I would encourage you to explore those next from here on out. But thanks and I hope you have a great conference and you find this information useful.