Hi everyone, I want to first thank the team from Conf 42 opportunity to speak and share our ideas with everyone. My name is Lionel Longcap. I work as an independent DevOps and cloud engineer and today I want to discuss with you how to build universal CI CD pipelines. Let's see our agenda. We will first start by giving you a little bit of history, how I came up with the idea, and then we will focus on the definition of some terms to follow up with what are the core principle of universal CICD pipelines. Then I will just give you what I just tell you what I learned throughout the process and we will just taking from what we will share together how you can go further with the same ideas a little bit about the history behind all of this. As a DevOps engineer, we are quite oftentimes tasked to create CI CD pipelines. Things to streamline the software delivery and to continuously improve the quality and the security of our software. The thing I realized throughout my experience after creating a lot of CI CD pipelines is that things are quite similar in the manifest we wrote and someday something started to shape in my mind. Let's see together here you have two manifests coming from two tools to do CI CD. The first one is a Jenkins file and the second one is the manifest file. The workflow manifest file from GitHub actions. I give you like one or 2 seconds to just see if something come up. When you see those two manifest, I hope you realize that when you remove the noise, we are only left with name of stages and actions and those two manifest files look quite the same. Let's take another example with a manifest file from GitLab CI same thing, stages and actions. So we could agree that if you have a way or a DSL to express only the intent, maybe a tool could be able to generate the noise and thus all those manifest files. I want to give you a disclaimer. Those ideas are free to use everywhere, so don't hesitate to copy and steal from those ideas because I really think that they will provide value. Let's see some definitions. I know everyone knows the definition of CICD. CICD stands for continuously integration and continuous delivery. Sometimes we use continuous deployment to include in CI CD, but those practices are very separated. But throughout the talk when I said CICD, feel free to think about the tree. So continuous integration, continuous delivery and continuous deployment let's see the definition of CI CD pipeline. A CI CD pipelines is a workflow first triggered by any means to automate every steps or activities involved in the delivery of a fit for purpose, that is functional requirements and fit for use nonfunctional requirements software. Sometimes CI CD pipelines look like this. You have a trigger. The trigger makes sure it's done. You come up with the source stage and then the build stage. We do some tasked here. Going forward to the test publish, we face the quality gate sometimes that just give us some guardrails before going live. So that's the simple look of CI CD pipelines. We are still on the definitions. Let's define universal in the dictionary. Universal mean applicable everywhere or in all cases general or used or understood by all. That's the two main characteristic of the word universal. This will help us define some core principle universal, as I said, used and understood by all and applicable everywhere or in all cases. I really like to change the first one by build Anubiquistos CI CD pipelines. If we get in the context of CI CD pipelines, and for the last one applicable everywhere, I will say platform agnostic. So a universal CI CD pipelines will be a pipeline that is built around a ubiquitous language that is semantically correct and that is platform agnostic. So we already seen this pipeline. So let's build one CI CD pipeline together, and not only a simple CI CD pipeline, but an ubiquitous CI CD pipelines. As I said, we start from the trigger, then after the trigger we went to the source stage. And if you zoom a little bit on the source stage, we realize that in the source stage we have many tasks to execute. We have sensitized the code, linting the code, analyzing the code, linting the docs. You have many tasked here. And after that we went to a stage that I like to call create. So in the create stage is where we compile the code, compile the docs, run some unit test, and then we went to the verify stage. In the verify stage, as you said, I tried to be expressive in the intent of the task here. So after verifying everything, we went directly to the review stage. And the review stage is where we take what we built and we put it in the QA environment. After that, and only after that when we realized that the package is right, we package all things up and we deliver it to an artifact repository. And I try to, as I said, be expressful or use good words and good verb to describe the intent here those to summarize that that's how ubiquitous CI CT pipelines will look quite expressive. And everyone that even know how things work could understand the pipelines without going into the implementation details. So after the delivery is where continuous integration and continuous delivery stop. And we have like an approval, manual approval that will forward us to another stage which is putting everything live and we have the same task at the end. So that's the whole idea. And continuous deployment only means that we remove the manual approval. So that's the UB crystals CI CD pipelines. To implement something like this, we need to have task managers, because task managers, as I showed you in those manifest file, task managers help just hide the implementation detail and let us express the intent. And also it's a good practice to use project generators because project generators help also to spread the good practice and make sure that everyone got the same templates and start from the same basis from the told requirements. My own preference as a tax manager, I use make and from project generators, I use progen. Progen is the project behind the AWS CDK and it differentiates itself from other project generators because other project generators got the same ideas. I take my template and I generate a project from the template. This works at the bootstrapping phase at the beginning of the project. But if the template evolve, you can't really get those updates in the project that derived from the template. So where project shines is that it keeps your template in sync with all projects that were generated from that template. Take a look at this project. It's very great then to build those ubiquitous CI CD pipelines, you have to shape your project and being a little bit disciplined about how you use your task managers. This example, I will show you how I use make to implement those, to make my task more expressive. So the action part, and there I have a make file and the make file is quite empty. The only thing is there is that I include all other make that present in the task folder. So I split every actions, every steps in those make files. And those make files are only generic because what the only thing they are doing is calling the script that is on the script folder. That's a little snippet of how to do it. And so I use this shape to produce my manifest. Let's take a look of a manifest from GitHub actions, because I really like GitHub actions. So you see, I just put my task and they represent the intent of what I need to do. And those instruction will forward everything to the script that could be written in any language. So you see that we have something that is quite expressive and everyone coming to the project without knowing the implementation details could understand the goal of our workflow. And that's the end goal of building very anubiquitous CI CD pipelines. The second principle of a universal CI CD pipelines is that it should be platform agnostic. So when you reflect on that, being platform agnostic will mean that we will have a manifest file that will help us define our workflow. And also you have to have something that will be well integrated into all the CI CD platform on the market and that will work on every platform. And what I realized that I found a real project that could do this, and it's dagger. Dagger is mainly a programmable CI CD engine. By programmable CI CD engine I mean that you could create not the entire manifest file, but the main part could be written using either Q go, Python Node js, and maybe a lot of other languages, because they created an SDK pal language to communicate directly with the dagger engine. And under the hood it works with Docker. So it ensure that your pipeline or the things you express using those SDK is also portable from CI to CI. Let's see, on an example you can scan the QR code to go to the project. You will see in this project something that looks similar to what I showed you here. I didn't implement any task managers. I just wanted to show you how I use dagger to program my CI CD workflow. So in the tasked you have there you have one task which is the script. Oh, this is my pipeline being programmed in Python. I won't show you the implementation because it's not important. For now, let me give you a walkthrough of the repo. So in the repository you have the GitHub, and in GitHub you have two workflows. We will see the CI CD workflow that is implemented without using dagger, and the CI CD workflow with dagger. To see how things are made and what the benefits we got, let me show you in real code how it's being implemented. Here you have a gif of those two workflow files. The one on the left is the normal workflow and the one on the right is the one using the dagger SDK or using dagger. So as I said, you see the differences between the two. We use an internal action called prepare, and its goal is to prepare the CI environment for both workflows. Let's see the content of this action. It's a composite action, and its role is to just install node JS. If we skip the dagger SDK, and if you want to use the dagger SDK, it install dagger IO and Python. That's it. And we also make sure that the version of the Python we are using in development is the same that is used in the CI. Let's go back to the diff you see there. That on the right is the workflow using dagger. The only thing that the workflow using dagger is doing is calling the Python script here. Let's see the content of things script. Here we import the dagger SDK and we're importing also any IU. Any IU will help run our pipeline. And the pipeline is just an async function. And inside that function you declare a client like this and you could build and customize a container that will be the base container for your action. So here I'm using the node 18 base image and I make sure that all my script run on that image here. And note that you could also do like multistage pipelines by also creating another container variables with the client container function. So that's all the things you can do with dagger. As you see, dagger is very powerful and you could have everything defined in pipeline script and the fact that there is a code. You can also imagine some use case where you can test your pipeline too. Now let's try those workflows on GitHub to see if it works. And let's hope that the demo gods are on our side. So if I go with actions normally, I would first execute the first action without dagger. So let me run the workflow, say run here. So the workflow will start and it will execute all the steps I showed you before. So for one of those steps, if I go directly to the workflow, you will see that it's very fast and the workflow was done. And at the end of the workflow normally what we got, let me go on here and we will see the package that were created after running this workflow. So if I go to packages, you will see that I have a new version published. So one 10 Rc eleven that were published 1 minute ago. That's a simple workflow. So let's use dagger then to trigger another workflow. If it works, we will have normally a new version that will be like one 10 Rc twelve. And there we are using dagger to do this. That's what I was saying, dagger very things there, because you can also create like multiple containers as code and just forward the output from one container to another one. And everything in the code, the workflow succeeded. And you see the workflow is simple. If you go to the packages normally, you will see a new version. Let me just refresh the page and voila, you have the twelve version. So that was Dagger and GitHub actions in the nutshell using the Python SDK. Okay, I hope you like the demo. Okay, what I've learned throughout the research on universal CI CD pipelines is that it's important to focus on semantics and less on technology and told, and also that onboarding is also important. So make sure that if you choose to write your pipeline in dagger to use the right abstraction to not confuse newcomers. And if you choose the tax manager route, make sure also, as we discussed at the beginning, to use good name for your actions so that they will just get the quick grasp of what your workflow is doing. The dagger team provides some sdks and I think that's great because you can come with the language you know and just start writing pipelines. I know today we have only go Q node js and Python, but the project is evolving so I think we will have many languages to come. Dagger could also replace in some part tasked managers, because everything is defined in your script as code and the only thing you have to do is call Python and the name of your script. So to keep your pipeline semantically correct, you could just choose to name your script the right way. Then what next? Today I try to follow the same process and my two exploration at the moment are universal project specification. So trying to define a universal project folder structure that could work with every framework, every language, any tool, and a universal code documentation specification that could help you create documentation and from that documentation create more generators and that documentation could be embedded in any language. Let's imagine that we have like something similar to Javadoc or GSDoc, but that will work with any language. So I hope you like the talk and feel free to connect with me with my provided handle. Thanks a lot for your attention and I hope that you will enjoy the conference.