Conf42 DevOps 2023 - Online

Manage Kubernetes addons for multiple cluster using runtime state

Video size:

Abstract

Kubernetes itself is not a complete solution. To build a production cluster, various additional addons like CNI or CoreDNS are needed. There are some solutions already. But if: 1. N Kubernetes clusters need to be managed; 2. the configuration has to change automatically when clusters runtime state changes; we feel a new declarative solution is needed. Given above requirements, Sveltos wants to figure out the best way to install, manage and deliver cluster addons. The idea is simple: 1. have one management cluster from where all managed clusters can be reached; 1. from the management cluster, selects one or more clusters with a Kubernetes label selector; 2. lists which Kubernetes addons need to be deployed on such clusters. Then have a controller that automatically detects new clusters and react to configuration changes and bring cluster states to match the desired states . Sveltos ClusterProfile This solution needs to be able to fetch information from the management cluster. For instance, when deploying calico, pod cidrs might have to be provided. Which pod cidrs has to be used in which cluster, is usually an information present in the management cluster (in a management cluster with ClusterAPI, such information is present in the corresponding Cluster instance, in the Spec section). Fetch information from management cluster So far, such solution still requires user to manage cluster labels. Ideally, which labels need to be attached to a cluster should be automatically detected by looking at cluster runtime state. This is especially true for long lived clusters. Let’s considered a very simple scenario. A set of addons (with a specific versions) are deployed in any production cluster running kubernetes version v1.24.x. When any of those clusters is upgraded to a Kubernetes version v1.25.x, a new set of addons (maybe just different versions) needs to be deployed. Instead of requiring user to change cluster labels when clusters are upgraded (which is error prone very easy to forget), we introduced a new concept: Sveltos Classifier. Using Classifier users can classify clusters based on: 1. Kubernetes version 2. Kubernetes resources and can define set of labels a matching cluster should have. Using a combination of ClusterProfile and Classifier, we can have Sveltos: 1. automatically detects cluster runtime state and updates cluster labels; - deploys new set of addons as soon as cluster is upgraded and its runtime state changes (because its labels have been updated by Sveltos as consequence of #1). Sveltos Classifier When deploying N addons in M clusters, an emphasis need to be on visibility. When making a change it must be clear what effects are going to be (which clusters are going to be impacted and how). So sveltos can be set in DryRun mode. While in DryRun mode, we can make any change and nothing will be propagated to managed cluster. Rather when done, we can use sveltos to see what would happen because of our changes and where (which clusters). Sveltos will create a summary table listing: - which helm charts will be deployed/deleted; - which helm charts will be upgraded (from which version to which version); - which kubernetes resources will be updated. After reviewing if fine with the change, Sveltos can be switched back to continuous mode and only then it will propagated changes to managed cluster. Sveltos also supports taking snapshots and rollback of the configuration.

Summary

  • Sveltos is a lightweight application that can be installed in the management cluster. It can manage kubernetes add ons in any cluster that is directly reachable from the managed cluster. Sveltos comes with built in support for cluster API.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello, my name is Marbente, I'm a principal engineer at Cisco Systems. I've been working on Kubernetes for the last five years. Today's talk is about Sveltos. Sveltos is an open source project that I developed which aims at simplifying kubernetes advanced distributions. When you have tens of clusters today, it's very easy for an organization to have multiple clusters and to have those kubernetes clusters in different cloud providers or infrastructures. And when you have that many clusters consistently managing addons, it's not a very easy task. Sveltos is a lightweight application that can be installed in the management cluster and that can manage kubernetes add ons in any cluster that is directly reachable from the managed cluster. Sveltos comes with built in support for cluster API. Cluster API it's another open resources project which is widely used to consistently create upgrade urban Kubernetes clusters. Sveltos as I mentioned has built in support for cluster API and that means that if Sveltos is installed in a management cluster where cluster API is also present, no action needed to be taken. Sveltos can automatically detect any cluster API public cluster and manage kubernetes add ons in such cluster. But Sveltos is not limited to cluster API. If you have a GKE cluster Anyks cluster and you want Sveltos to manage add ons in such clusters, you can easily register this cluster with Sveltos and from that point on Sveltos can seamless manage kubernetes add ons in any cluster. I'm not going into detail of how to register a cluster with Sveltos. It's a very simple procedure which is listed in the Sveltos documentation. So please refer to that one. What I want to highlight here is that any cluster it's represented in the management cluster. So cluster API power clusters represent the management cluster by the cluster API cluster CRD any cluster which type GKE which is registered with Sveltos it requirements a management cluster by Sveltos Cluster CLD instance. So how does Sveltos solve this kubernetes add ons distribution? Those idea is simple. Select a subset of the cluster that we manage and list the add ons that we have to deploys in such clusters. In this example I have a cluster profile inserts. Cluster profile is one of the Sveltos crbs and the cluster profile spec section has two main solutions. Those is a cluster selector and there is an l chart section. Those cluster selector it's a pure Kubernetes label selector. In this case this cluster profile. It's telling us Sveltos to selects any cluster that matches that has the label environment production and it is also this cluster profile. It's also instructing Sveltos to deploy Caverno L charts version 2.5.0 in any such cluster. So what happens if we add those label environment production to the two workflow clusters? Well, Sveltos automatically detects that those two clusters are a match for this cluster profile because this cluster profile is saying that the Caverno charts needs to deploy those sveltos automatically deploys caverno in but clusters. Now one thing which is important is that if you are when you deploy an elm, shark or Kubernetes resource in many clusters, very likely you need to customize the helm shard per clusters. For instance, if you are deploying calico, you need to tell Calico what the podcider is. In this example I have a cluster profile which is instructing Sveltos to deploy Calico and it's also instructing Sveltos to fetch the podcider from those management cluster at runtime. Runtime means at the very exact time when Sveltos is deploying calico in a cluster, fetch that information. And in this case Sveltos is fetching that information reading the cluster instance that represents the cluster and reading the spec cluster network cyber blocks fields. Now let's go back to our example. We had Sveltos deploy Caverno in any cluster with those label environment production. What happens if now one of the admin in one of the managed cluster detected the Caverno deployment. So we have one of the admin issue in the Kubectl delete deployment and what happens that Caverno gets deleted from the managed cluster. But if we do that now we have a workflow cluster state. It's different than the expected state. The expected state is what has been defined in the management cluster. The expected state or any cluster with label environment says that Caverna needs to be present. As of now Cavern has been deleted. So Sveltos has a configuration bring detection configuration bring detection means that there is can agent that Sveltos installs in each managed cluster and this agent has one very simple prone watch any resource which is deployed by Sveltos and if this resource is modified, notify immediately Sveltos running the management cluster. In this example, Caverno was one of the resources that Sveltos developed deployed in the managed cluster. But this resources has been deleted by issuing a command directly in the managed cluster. So the agent running in this managed cluster automatically immediately detects that there is a potential configuration grid and it informs Sveltos running the management cluster. As soon as Sveltos is informed, Sveltos reconciled back and by reconciling back, Caverno gets deployed back. The state of the workload cluster goes back in sync with the expected state defined in the management cluster and there is absolutely no admin intervention that is needed. In all the example we have considered so far, we has the management cluster admin set the labels on the clusters, but we are talking about managing tens, potentially hundreds of Kubernetes clusters, so we cannot expect those management cluster admin to be the only one in charge of setting the labels on each clusters. I wanted to have like a more intent based mechanism for that. I mean that I wanted to be able to say if cluster runtime state matches this, I want this set of labels to be automatically added to the cluster by Sveltos. Let's take an example. So we have a cluster profile which we can ignore for now. And we have a classifier classifier. It's another Sveltos CRT and those classifier in the spec section saying it has a Kubernetes version constraint saying if you see a cluster running Kubernetes version between prone 24 and strictly less than 125, such a cluster is a match for this classifier. And any cluster which is a match for this classifier needs to have those classifier labels target. In this case the label are simply gatekeeper v three nine. So let's see what happens. In this example we have two clusters whose version is 124. Two. So they are both matching this classified. And because they are both matching this classifier, Sveltos automatically adds the label gatekeeper v three nine into both clusters. And when we use a classifier in combination with the cluster file, because now those labels, those clusters have the label gatekeeper v three nine, they are a match for this cluster file. And because they are a match for this cluster profile, gatekeeper gets deployed in the cluster. What happens now if we post another classifier instance? Classifier instance it has a different Kubernetes version constraint saying that any cluster running a Kubernetes versions greater or equal than 125 is a match for this classifier. So what happens if we upgrade one of the clusters? If one of the cluster gets upgraded, it stops. Bring a match for the old classifier and it starts. Bring a match for the new classifier. Because a match for the new classifier, the labels is updated to gatekeeper v three because the label is updated. Now the classifier matches different class of file and because the classifier matches different class of the file, now a different version of gatekeeper is installed. So the gatekeeper gets upgraded from three 90 to 3100. So combination the classifier with a cluster profile. We can automatically have Sveltos update the cluster labels based on the cluster runtime states and when the cluster labels change those cluster stop. Matching cluster files has matching different cluster files so those set of addons set gets deployed in the cluster changes classifier. It's not limited to classify a cluster based on the Kubernetes version. You can classify a cluster based on any resource. There are many examples that you can refer to in those Sveltos documentation. Sveltos also has support to take snapshot. Snapshot is another Sveltos CLD and in this case this snapshot instance is instructing Sveltos to take snapshots every hour. When Sveltos takes snapshot, it takes like a snapshot of the configuration in the managed cluster. Once you have more than prone snapshot, you can use Sveltoscaddle which is a CLI that I developed explicitly for Sveltos and you can take a look at all that has changed between two snapshots. And if there is something that you don't like, Sveltos has also support for rolling back the configuration. So you can start Sveltos to say you can tell Sveltos, bring the configuration back to this snapshot instance and Sveltos is reconciling the management cluster configuration to the snapshot instance. Sveltos has many other features Project Sveltos IO it's the website with the full documentation and there are many examples on how to register a cluster how to classify a cluster based on resources top of versions GitHub.com projects Sveltos is where all these Sveltos repos are and as I mentioned, Sveltos is an open source project. Any contribution is welcome. So if you have an idea, if you want to contribute, if you want to use it and you have any question, please feel free to reach us on slack projectspeltoslack.com and I will be happy to clear from you. Thank you so much. Thank you for your time. I hope you found this presentation useful and again, I hope you find the a problem and the solution also a good solution. Thank you so much.
...

Gianluca Mardente

Principal Engineer @ Cisco Systems

Gianluca Mardente's LinkedIn account Gianluca Mardente's twitter account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)