Hello, everyone, and thank you for joining me today at Conf42 DevOps. My name is Andrea Andriushan. I'm a DevOps engineer at Analog Devices, and I'm excited to talk about how we've optimized the release process for Analog Devices custom Linux distribution. Named KyperLinux. The software development life cycle is all about improvement. Iteration by iteration. Every stage, including the release process, benefits from refinement. When done effectively, this ensures that products reach customers faster, with higher reliability. Today, I will show you how we applied this principle to KyperLinux and the substantial results it brought. Now, you may ask, what's our custom Linux distribution? Well, Let's answer this question and tell you about ADI KyperLinux. Kyper is the primary distribution for product evaluation boards, including reference designs, device drivers, and a variety of development utilities. Now, let's see how we improved the release process on this complex component. By the end of this talk, we'll gain insights into what was the old release process for Hyperlinux in order to have a better understanding of the changes that were made. and how they impacted the product and which are some of the major improvements that we needed to integrate and why. And also the current state of our custom Linux distribution. The traditional process was slow, rigid, and prone to delays due to lack of modularity and the high interdependence between components. All the software components were built during image creation, which meant a longer build time. Another reason for the long release cycles was the fact that software components were released through Kyper Linux image. With no modularity, a bug found in one of the components delayed all of them. After we identified the big problems of the previous flow, we came up with a few ideas that we then implemented. These key modifications were creating a stabilization branch from release of each core software component, breaking down the monolith structure for a better management of the Linux drivers, libraries, UA application, and user space tools. And creating Debian packaging for components that were previously mentioned. Modular configuration for custom images. And based on the other changes mentioned, we can configure easier the image before build time. expand component level testing with custom docker images created from a Kyper Linux build image. Improvements were also made on the CI side by moving the automatic build on GitHub Actions and introducing PR templates and co donors for a better overview of the new changes. To kick off the improvements, we introduce a dedicated stabilization branch for every new release. This branch is created from the main branch and serves as a space where the release is refined, tested, and stabilized without interference from ongoing development. This approach allows us to focus on preparing the release while the main branch continues to accommodate new features and updates. The result? Faster issue resolution, more focused testing, and a stable release pipeline. Based on the diagram below, you can see that the master branch serves as the main development branch for new features and fixes. Our next stable branch is created by branching off from master. This branch is dedicated to stabilization. Ensuring that the code is closely tested and prepared for release. Fixes and updates are often pushed to both master and nexttable simultaneously to maintain consistency. Once the nexttable branch is considered ready for release, a release branch or tag is created from it. In most cases, new bugs are found during release testing. That's why a release patch will be created after the fix is added on the branch. After a release is finished, the next stable branch is overwritten with the latest changes from master to sync and prepare for the next stabilization cycle. Next, we moved away from a monolithic release process and separated the entire release into smaller independent components. In the past, everything was bundled together and a bug in one area could delay the entire release. Now, components like software tools, libraries, and boot files are managed independently. By breaking down the release this way, we've eliminated unnecessary interdependencies. Each component can be developed, tested, and released on its own schedule, leading to faster iteration and reduced risk. For each software component, Debian packages are created as a release artifact. After that, they're uploaded to our Linux package repository. By using standard Debian tools, we've simplified distribution and made it easier for users to install or roll back software packages. But one of the most significant steps we've taken to increase flexibility is the introduction of a modular configuration file. This file allows users to build custom images by selecting the specific tools they need. And even specifying the version of those components. For instance, If you're working on a project that requires a specific library version, you can specify that in the configuration file before build. Additionally, we've made it easy to introduce new components or additional dependencies. With the help of an extra script, users can add other compatible tools. This approach not only makes the release process more modular, but also gives users the freedom to craft their ideas. images with their exact requirements, ensuring that no unnecessary components are included. Now, to improve reliability and compatibility, we've implemented our robust testing process that works at two levels. First, each component is tested individually to ensure its stand alone functionality. This allows us to quickly catch and address issues specific to that component. Second, the components are tested as part of the full Kuiper image, but with a twist. We've introduced lighter Docker images that are built with the specific dependencies required for each component. These Docker images replicate the Kuiper environment during the development process. By testing within this controlled environment, We can, we can ensure early compatibility with the final system. This approach allows us to catch integration issues during development, rather than later stages, which saves significant time and effort. And if you want to know more about our testing process, my colleagues have two presentations about it at this year's edition of Conf42. Thank you. Bianca is presenting a secure integration of private testing infrastructure with public GitHub repositories. And Stefan's topic is Advanced Test Harness Infrastructure for Validating ARM and FPGA based systems. To optimize our CI pipeline and make the development process more accessible and efficient, we transitioned from Jenkins to GitHub Actions. This move brought several benefits. First, BuildLogs are now openly accessible to all users. increasing transparency. Additionally, fresh build images are automatically generated and made available on GitHub as soon as the commit is merged into the Kuiper repository. This eliminates the need for users to wait for an official release or manually build the image themselves. We've also expanded the build configurations. Previously, we only built a 32 bit full image. Now, we've added support for 64 bit architectures and introduced both basic and full configurations for each architecture. This flexibility ensures that users can choose the image that best suits their needs. To further improve collaboration and code quality, we introduced a pull request template. This provides a consistent structure for a contributor to describe their changes. Additionally, we've implemented the co donors feature in GitHub. This ensures that relevant team members are automatically assigned to review and approve pull requests, speeding up the review process, and maintaining high standards for the code base. These changes make the build process faster, more flexible, and more user friendly. while also encouraging better collaboration among developers. Now, let's take a closer look to the current state of our release process. By using a configuration file, users can now create custom images mapped to their specific requirements. They can select only the components they need, ensuring lightweight and efficient builds. We've also shifted to build each component independently and releasing it as a Debian package. These packages are uploaded to analog devices Linux package repository, which allows for faster and more flexible builds. This change also simplifies updates for users, making, making it easier to install or rollback components as needed. By dividing the release cycles. of individual components, we've ensured that bugs in one component don't block their list for others. This approach simplifies the overall process, minimizes bottlenecks, and increases development efficiency. Now, in the end, thank you all for listening, and if you have any questions regarding the presentation, or if you need more details about it, feel Do not hesitate to contact me on email or LinkedIn. And thank you.