Are youll an SRE a developer, a quality engineer who wants to tackle the challenge of improving reliability in your DevOps? You can enable your DevOps for reliability with chaos native. Create your free account at Chaos native. Litmus Cloud hello everyone. Welcome to this session. Deploy n applications to n clusters using Argo City application set my name is Dewan Ahmed. I'm a developer advocate at Red Hat. I'm from Moncton, New Brunswick, which is in the beautiful Atlantic region of Canada. I love everything cloud native with a focus on DevOps and Gitops tools. Besides work, I love to play pool and ping pong. I'm also a freelance carrier coach, where I help students and new grads to start a career in tech. In terms of today's agenda, we'll start with the idea of Githubs, especially Githubs on Kubernetes, and then we'll cover the popular open source tool Argo CD. The main focus of today's session is a subproject of Argo CD called Application set. For both Argo CD and application set, we'll have demos and at the end we'll have some time for Q A. On the bottom right corner of your screen youll can see a link to a Git repository. In that repository I have added detailed instructions on how you can do demo on Kubernetes platform. The demo I'll be showing will be based on OpenShift platform. If you have a mobile device, you can also can that QR code which will take you right to that git repository. Luis Fasira summarized Gitops in one slide. So Gitops in one slide means there are three main principles. We use Git as the single source of truth. We use Git as the single place where we operate. That means not on direct clusters, but we do that on git, and the challenges is reflected on the clusters. All the changes we're making will also be observable and verifiable. From the fun fact of Gitops Alexis Richardson first came up with the term Gitops, and when he ran it by his friend, his friend thought it's the ugliest word he has ever heard, and now he can't unhear it trying to come up with a term that is easy to pronounce but also difficult to forget. Alex knew that he came up with a perfect term. Now let's understand Gitop's BoM or bill of material on kubernetes. First we need Git repo, where our application manifests files will reside. Then we need kubernetes cluster or OpenShift cluster where we are deploying our application. You might also need a manifest generation tool like customize. If you have different environments like dev, Preprod and Prod, don't use a CI server to orchestrate straight direct updates to kubernetes as a set of CI jobs. There's a blog post mentioned in the resources section that covers why this is an anti pattern. And finally, you'll need your kubernetes manifest files such as deployments and services what is Argo CD? ArgoCD is declarative, GitHub's continuous delivery tool for Kubernetes. It is part of an open source project. The project started at Intuit, but then companies like IBM and Red Hat and various individual contributors have contributed to this project. From this diagram, the concept is pretty simple. ArgoCD watches over your Git repository and whatever change you have or whatever configuration you have in your Git repositories, ArgoCD implements that change declaratively on your clusters. Now why Argocd? The first few points have been covered in previous slides, like why we're using that. In Argo CD, you use git push for managing your application, delivering including changes to your cluster. This is the flow your developers are familiar and comfortable with. In terms of security, Argo CD offers so integration based on OIDC or Auth two, and it offers a single sign on for various social profiles as well, including GitHub, GitLab, LinkedIn, et cetera. In terms of automation, ArgoCD offers multi tenancy and arbic policies. ArgoCD also offers audit trails for application events and API calls. In terms of rollbacks or youll anywhere, it's a matter of youll application configuration committed in a git repository, so you can go back to any specific commit that's in your git repository, either using the UI or the CLI. If you do Argo CD app diff from your CLI, ArgoCD performs a diff between target and a live state of the cluster, and this can also be seen on the UI, so you can observe the drift of your application defined in git and the state in the target cluster. Now considering all this, your application team will have a higher velocity for faster application delivery. Now how Argo CD works Argo CD is implemented as a Kubernetes controller which continuously monitors running applications and compares the current live state against the desired target state as specified in the Git repo. So on the left hand side you can see our developer Persona who does a gate push or after a pr merges, ArgoCD reacts to that change. A deployed application, these live state deviates from the target state is considered out of sync. The ArgoCD API server is a GrPC and rest server which exposes the API consumed by the web UI, CLI and various CI CD systems that we can see on the left hand side of our screen on the blue square you can see there is a repositories server which is an internal service that maintains a local cache of the git repositories holding the application manifests. Synchronization can be configured using resource hooks so that what we see at the bottom of our screen hooks are ways to run scripts before, during and after a sync operation. One example or use case of such hooks is if you want to orchestrate a complex deployment that require more sort of sophistication than the Kubernetes rolling update strategy. You can have a script run before, during or after the sync operation. All right, so that's enough theory. Let's dive into Argo CD demo hello everyone, the first demo is on Argo CD. This is based on OpenShift, but if you'd like to follow a Kubernetes based demo, you can go to this GitHub repository and follow the instructions. This is based on vanilla Kubernetes. Coming back to OpenShift, we are installing can operator called Red Hat Openshift Gitops operator. This operator out of the box gives you an argo CD instance and also installs applicationset controller under Openshift GitHubs namespace. To begin with I'll connect to the argocd UI. For that I'll need the default argocd admin password and we can find that under workflows secrets under Openshift GitHubs project. Openshift GitHubs cluster is the name of that secret. I'll copy the secret and open Arcocd UI from here. The username is admin and password is what I just copied. Arcocd has a concept of projects. Now this is not Openshift projects. This is Arco CD projects where you can group your deployment applications. You can also add repositories. If it's a private repository, you can connect using your password or token. The repository which we are using for this demo is a public repository, so we don't need to connect it in advance. So let me copy the URL of the repositories and try to create an application. You can create application from the arco CD UI or using arco CD Cli but I'll create the application using the OpenShift console. So under the operator I'm going into application create application, let me give it a name. Argo sample app and then under destination I have to give the server location since it's the same server where arco CD is installed. It's the default Kubernetes service. And the namespace I'll use Openshift gitHubs. You can choose deploy in a different namespace and you have to add a label that says managed by Openshift GitHubs. But for now I'll use Openshift GitHubs namespace project is the argo CD project which is default. That's the only project that's on these Argo CD server now. So only default project. So that's what we are using now under source I'm adding the repository URL which I just copied and under path customize nginx openshift so this is the path where my manifest files are for sync options or sync policy, which basically says that if youll like to automation sync the application when there is a change in git repository, or youll like to manually sync either clicking the sync button or using the cli I won't go with the automated route now, so it's a manual sync option and I'll create these application. Once I hit create on the Arcocd console, I can see the app was created. If I go inside I can see that my three resources, the service deployment and these route were created. And if you look at the automation file you can see that there was a name prefix prod which you can see in all three resources as well. Now once I sync the application, you'll see that the applications will sync and it turns green. It's synced sync ok, and you see the single pod that is being created. Now if I'd like to change these number of replicas from one to these, let's see if I make a change. So going the git ops route, making change to the git repository and seeing the change reflect on the cluster, I'm not going to go through the pull request route, I'll push directly to the main. So once I do that, if I hit refresh, I'll hover over here, see the synced options should turn yellow out of sync. So it's out of sync now and I can click on app diff to see the difference what changed. So when I scroll down you can see on line 110. So these number of replicas in these cluster it's one, but in get repository expected or desired is three and hence the application is out of sync. All we need to do is hit sync, synchronize and the application will go in sync. All right, so we can delete the application either from here argo CD UI, or we can delete the application from Openshift console as well. Let's delete the application. I can create the same application, but let's use auto sync now. So I'll say this is arco CD sample app. Let's say this is autosync destination, the same cluster and namespace, same ergo CD project under source I'm using the same source that we used customize nginx Openshift and then under sync policy I'll choose automated and then hit the prone and cell field. What prone does is if you delete a resource from git, then argocd will delete that on the cluster as well. And if you check self heal, then even if you delete a resource directly on the cluster, Argocd will not let your app go out of sync and it will prevent youll app from even seeing that challenges. So let's keep those two options and create the app again. And if we go back to application, so the new application was created and this time, if you notice, we didn't have to click sync because it's auto sync, these app was automatically synced. So let's try to delete one of the resource to see the self field feature in action. So for that I'll log in to the cluster. So let me log in and I'll switch to openshift Githubs. I'm already openshift GitHub's project, so let me check the service. OC get service what are the service I'm getting? Okay, I see the prod nginx, so let me delete that service. OC delete service prod nginx and then observe what happens on Arco City UI. So if you observe here, you don't see the app going out of sync. And that's because even if we try to delete that resource, which you can see, the service was in fact deleted, but it got recreated. Even if you refresh, you see the service is right there. So Argo city prevented that operation from happening. So that's a pretty powerful feature. With auto sync and cell field, we created the app using OpenShift console. You can very much delete the app from Argo CD UI. It's using the same set of APIs so it will be the same operation. And that's the demo with arcocd deploying a single application from single git repository to these same cluster. In the next demo we'll see arco city application set where we deploy application to multiple clusters. That was a demo of ArgoCD. These we deployed one application from a single gate repository to a single cluster. But what if you youll like to deploy Argo CD application to multiple Kubernetes clusters all at once? Or what if you had multiple argo CD applications in the same repository? More like a Monorepo introducing applications set controller for Argo CD unlike with an argo CD application resource, which deploys resources from a single git repository to a single destination cluster or namespace, applicationset uses templated automation to create, modify and manage multiple ArgoCd applications simultaneously, which targets multiple destination clusters and namespaces. The applicationset controller is installed alongside ArgoCD within the same namespace, and it automatically generates ArgOCD applications based on the contents of a new application set custom resource on the right hand side of the YAML on line number two you can see the kind bing application set. Now generators are responsible for generating parameters which are then rendered into the templates field of the application set resource. We'll be seeing two of the generators in the demo today. In this example, the list generator passes the URL and the cluster fields into the application template as parameters, which are these rendered into the template as three corresponding ArgoCD applications, one for these each defined cluster targeting new clusters is simply a matter of adding new deployments to the application set resource, and the corresponding ArgoCD applications will be automatically created. The sole responsibility of the applicationset controller is to create, update and delete application resources within Dargo CD namespace. The controller's only job to ensure that the application resources remain consistent with a defined declarative application set resource, nothing more. Thus, the application set controller does not create, modify or delete kubernetes resources other than the application custom resource does not connect to cluster. You have to already have the clusters defined in Argo CD application controller does not interact with namespaces other than the one where ArgoCD is deployed. It is Argo CD itself that is responsible for the actual deployment of the generated child application resources such as deployments, services, and config maps. The application set controller can thus be thought of as an application factory, taking an application set resource as input and outputting one or more Argo CD application resources that correspond to the parameters of that set. Here are some of the generators for application set. You can find more details of each of the generator from application set. Read these docs page, which are in the resources section of the GitHub repo which I showed earlier. All right, again, enough theory and let's dive into application set demo. In the second part of the demo we'll use argo CD application set to deploy applications to multiple clusters. Let's look at the example. In these examples we have some generators, but specifically we'll use list generator and git generator directory example let's start with list generator. Now in the list example we have a custom resource with kind application set, and we're trying to deploy this particular application, argo city application, into two clusters, engineering dev and engineering prod. I'll be commenting out this engineering prod resource pretty soon now. One thing to notice is the template part is pretty similar to an arcocity application, and here some values are dynamically injected. For example cluster. These values would be coming from here and here, and URL value is coming from the URL files. One other thing to notice is if you don't have a namespace, you can use this flag create namespace equals true for the remote clusters. The first thing to do when you're deploying to multiple clusters is adding the remote cluster to Argocd server. First, I'll log into ArgoCD server. Let me go to server by clicking this link, which will open up the ArgoCD server UI, and we'll use the default admin account with the admin password to find the admin password. If you're using Openshift, go to workloads secrets from Openshift GitHub's namespace and you can get the Openshift GitHubs cluster secret and that will get us logged in to the cluster from here. If I see clusters, I can see there's only a cluster. That means where ArgoCD is installed, there's no remote cluster added. In order to add the remote cluster, we have to add it through Argo CD Cli. You cannot do that from argocd UI. So let me log in to argo CD server from the command line. I'll do argocd login, and argo CD is pretty picky about the URL, so I have to remove the HTTPs part, the username admin, and the default admin password. All right, so once you are logged in, we have to log into the remote cluster. So this is the remote cluster. Let me see if I'm already logged into the remote cluster. Yes, I'm already logged into the remote cluster as you can see. So we have to add this context argo CD cluster add and these context. So once we do that, a service account cluster role and cluster role binding will be created into the remote cluster. That will let Argo CD manage resources in the remote clusters. I already executed this step beforehand, that's why it says it already exists. Let's start with the list generator example. In this yaml we have an application set custom resource called Guestbook and we'll be deploying this application in two different clusters. One is the cluster where Argoc is installed and the other is a remote server. I'll uncomment these two lines shortly, so let's copy this yaml you can create this application set custom resources from these CLI, but if you're using Openshift Githubs operator, you can create this application set from the console. So we hit create application set button and we paste the custom resource here. Now one thing to notice is because you're creating this application set in a different namespace than Openshift Githubs guestbook ns you'll have to create this guestbook ns namespace beforehand with a specific flag called managed by Openshift Gitops. So here's that namespace youll have to create this beforehand. Notice this specific label field. So I have already done that. I'll uncomment these two lines and also update the URL field. So let me go back to my argocd web UI and from cluster I can see the remote cluster and I'll update this value which is engineering prod and let's create this application set. So once I create that under application I can see these two argosd applications are being created, one in the cluster where argosd is deployed and one is in the remote cluster. Both apps are synced and healthy. If I try to delete one of these applications, you'll see that the applications will be out of sync shortly. But applicationset controller will recreate this application because these applications were created using application set. In contrast, if you created these two applications separately using Argo CD, youll be able to individually delete this application. So that was the list generator example demo. I have to delete these two applications from here using application set and now I'll be able to delete this application. Now let's look at the git generator directory. Example for the git generator directory I have three different applications in the same repository, a helm, a jSonnet, and a customized application and similarly I have to create three namespaces beforehand because these three applications are being deployed in three different namespace. So this specific label allows Openshift, Githubs or argo CD to be able to create resources in those namespaces. I have already created these three namespaces. Now let's look at the example for git generator directory. I'll create application set from the Yaml view and here you can see that path base name which is all the folders in this specific path, which are the three applications guestbook helm, guestbook, JSON and guestbook customize. These will be the three arcocity applications that will be created. Let's create this application set and we should see three applications will be created. I did not have autosync enabled for these three applications, so I'll have to manually sync this application. Once I click on sync, these three applications should be synced similarly to the previous example. If I try to manually delete any of this application argosage application set will create this application again. So that was these demo for git generator directory and you can practice on your own on these generators or if you want to test out matrix generator. That example is in this repo ergo city application set demo and if you scroll down to the bottom of the page you can red hat. I have an example for matrix generator as well and this repository is based on vanilla kubernetes. To sum up in this talk, we learned about the concept of Gitops, more specifically Gitops. On Kubernetes. We talked about a declarative Gitops tool for Kubernetes Argo CD. We deployed an application from a single repository to a single cluster and then mentioned about the challenges of deploying multiple application to multiple clusters. We talked about Argo CD application set which is a subproject of Argo CD, and how we can use various generators within application set to deploy n number of application to n number of clusters. Thank you for your time today to listen to my talk and I welcome any questions you have.