Um, hello everybody, welcome to my talk in Conf fourty two this year on the DevOps tracks, we will talk about how in my favorite banks, we made these reality to implement the it controls, especially the security controls over the complete release supply chain, whatever the technology, whatever the teams, whatever the departments, because some of them may have some different released processes, especially because of the technology, but also because of what they know and what they've done so far. But at the bank level, some controls, some security checks are defined, and we make sure that these controls are a reality. So I will start with a small presentation. I'm Manuel. I've been working in these DevOps world since long time, even before the term DevOps did exist. I worked in several companies. You see the logos, the last ones are electric cloud or cloudbees or Wipro. And I'm also a DevOps Institute ambassador where we talk and think about the human aspect of DevOps, which is very important in the project, as we will see today in that lessons learned oriented talk, a little bit of history. Just to start, we will not go through everything, obviously, but over the last 25 or even 30 years, which is roughly the time I've been working, there has been a lot of changes in the infrastructure in these application architectures. That leads to a certain number of challenges. Okay, roughly, if we want to sum up at the maximum, we've switched from the monolithic application to modular application, whatever it means. Today, modular means microservices oriented architecture, and it brings, I would say, a little bit more of complexity in the release process itself. So roughly, if we look back in the past years, we've had several waves, languages, method, process or product. And then now DevOps. Each of these waves has been characterized by some capabilities oriented on languages, oriented on the first case, tools for constructing software systems and monitoring. Then came the CMM, it and agile. And now we are talking about DevOps, about pipelines, about automation, obviously applied to the application. One of the main characteristics that comes from the state of DevOps report is that the deployment frequency recently grew a lot, really a lot, because mainly of the microservices application microservices architecture that intrinsically brings a lot of capabilities in releasing fast, releasing fast, mean release frequently. So when we talk about DevOps tooling that has matured, we roughly at the very beginning switch from a job based release orchestration, release management to a pipeline based for a second generation. So we had some scripts, and basically what we have done is that we have wrapped these scripts into some stages and environments. Obviously, even if it was a quite poor automation. We had the ability to define some access rights, for example, which is a very first important step, second to third from pipeline based to adding a lot of additional things, testing for example in the automation process. So stages and environment to QA CI, a little bit of release automation, starting with some best practice, the fidelity across the environment, the inventory, know where we are and what we do, the change management. Why should I start from a point a to go to a point b and requirement management. What are the link between what I'm doing and some requirements? So this is where Jira grew up also to make the link between all these aspects of software delivery. And obviously because this is one of the purpose today, add more security. Security in terms of security of the software and security of the process. So approvals, know who did what. Audit trail. All these security aspects that grew up between my second and third phase, between third and fourth phase, where we are today, a fully configurable system where we can have scalability and ease of onboarding for new applications. Very often we have some model based tools, model based, that allow us to switch from the simple pipeline to a proper DevOps platform where we decouple the deployment activities from these release activities. So what does it bring? It brings a complete audit trail with the approvals, with the ability to automate the approvals and all the reporting and dashboarding that I can imagine on all the data I'm gathering. Self service. It's ease of use for the consumers, for the new projects that are onboarding on these platforms. Roughly one, two, three and four. When I think about the evolution of DevOps, I'm also thinking in terms of enterprise scalability. And I've gone through some challenges that I still have to face today. Okay. Challenges in the financial services world, where I work mainly, and we will focus today on the third one because it's devsec abstracts. Security and compliance. Security and compliance. I go through some organization potential problem. Sometimes there is one centralized tools group that makes decisions which are not necessarily linked with the field, people that actually do the development or the operations. But I also have to deal with the platforms. Several clouds, several technologies. Am I fully kubernetes? Do I use Openshift or Tanzu or rancher? And each business unit that may have its own way of working with a release team, with a specific technological stack, with some licenses and with some different processes. So it's not always easy to deal with all that and to define something that would make everybody happy. Okay, you recognize here the wall of confusion between development and operation in the background. So the challenges, I will not go through this slide because it's unreadable, because I have too many challenges. Challenges in terms of spending, in terms of visibility and in terms of morale, the business loss or the people lost also. So in order to try to address all that and all that can be quantified, okay. It's quite easy to have the ability to put numbers for spending, for lack of visibility, for lack of morale or business losses. And at the end of the day, we can calculate, and this is what we will do in a couple of slides, a return on investment. When we think about another solution, a new approach that will allow us to have a better visibility, have a better morale of the employees, have a better flexibility, and to really scale, okay? Which also has some impacts. Impacts in teams of spending, in terms of overtime, I will spend more time, I will spend less money in hardware or in spending in my different clouds and in efforts, because I do not have to maintain a homegrown or some homegrown solutions. When I define based on some market tools that can be open source or not, it's not the question, but if I choose something that I can rely on, I can, but the effort to maintain these on ground tools. So my return on investment, what is it? For the very example that I'm talking about today, the return on investment, it's been quite easy to calculate. You have the cumulative spending investments in red and the benefits in blue. So we are with a fair case. It's not the best case. It's not the worst case. It's a fair case. And you see that we have the return on investment after five months of fully using a new approach, a new production that will allow us to have that transversal matter in terms of security and implementing of devsecops. So what does it mean in terms of processes? Okay, so I focus on the security pipeline. This is not the only pipeline which is transversal, but this is our topic today. That security pipeline is roughly the answer to several questions. I want to do CI CD, I want to be compliant. I want to build a bill of material. The bill of material. We will go back on that in several slides. Joe Biden has published an executive order on the bill of materials. So it become very important to be compliant. But it's not my focus. If I am working on a project, it's not my focus. I'm okay to be compliant to these security standards, but it's not my job. My job is to develop applications and to make sure that these applications while verifying these security standards will work in production. So I'm doing DevOps, but security is something that I want to complies to, but it's not my job to define how I have to be compliant. So basically what we have defined as a DevOps team, we have defined a security pipeline made of several stages that we have agreed with the security people, where we have to comply to several stages, several tasks in the stages for development, for integration, for delivery, for pre production and production. We have to do some scans, we have to verify some security aspects. So let me introduce these generic checks and controls. The important thing is that they cannot be changed by me. I am working on the project, so I don't have the choice to go or not go through that security pipeline. I have to be compliant. I cannot bypass these checks. They can change anytime, even between two deliveries. We will go through an example of that, which was a workshop I did some time ago, which is these very explanation of this change that can happen anytime. They are flexible enough to cover all the cases for the technologies, for the process itself. So a huge number of parameters and they are open and scalable so they can embrace new tools and new technologies. Let's focus on that security pipeline. So it's made of examples, the examples we have made for this implementation, the checks could be other ones, it could be other tools rather than Sonacube or checkmarks or the IQ server. But roughly this is what we go through and we will see how we feed back that information in the main application delivery pipeline. Just a small focus. The stage of development, we have to go through some SaaS analysis with three different tools. But before doing that we ensure that the user has gone through the training, so he knows what he's doing and he knows how to analyze the feedback he will have. And at the end I have an exit gate, so I will block the delivery if the thresholds for these three SAS analysis are not met or are not satisfactory. Same thing. Another example for the preprod deployment. I verify that I have gone through these checks, so a dust and some of the different checks. And at the end of these day I will verify that my preprod rules are okay in order to allow to move forward. So my security pipeline will be used by an application pipeline. Remember I am the application. So once more I still want to do CI CD, I want to be compliant. I want to build my bill of material using the security checks that has been defined over there and building my bomb and applying these internal controls for compliance and with built in security. So what does it mean? It teams that you remember my pipeline for the security five stages. My application pipeline may have some very different stages, okay? It has been defined like this in my department since years. So I like to have these environments. So what I will do, and this is what we've done, we just synchronize things. We synchronize saying, for an example, okay, to start the QA stage here on my application, I have to wait for the results of the integration checks in security. And if these integration checks in security are okay, you remember my conditions for passing the gate. If I can pass the gate out of integration in security, then I can move to QA and deploy to QA and do my QA test, and then after that my sight system integration test and so on. So I synchronize the two pipelines, the application that can change and the security that never changes. An example of application release pipeline where this step start, my security pipeline cannot be removed, cannot be bypassed. You remember, I have to go through that, and it will make sure that I synchronize everything afterwards. So, well, my pipeline is quite big. I have several checks that are related to my application. I use my own ticketing system. For example, in stage seven, I check the bomb entry, the CMDB entry, the mutability. These are things that are related to my application. But in the meantime, in parallel, I will have all my security checks. The example, if I do a small focus, what happens in development and the conditions. User has gone through the training to enter once more and the gate rules to get out. I have to have the same threshold, okay, for the different analysis, same thing for the production here. So it becomes serious. I'm switching to production. I'm handing off. I have to make sure that I have a bill of material. I have an entry in my CMDB because this is the way my favorite bank works. I have to check the immutability of that information. And I have to have an SKU, because any application that goes into production has to have an SKU in my case. But you can define all the rules that you want in order to condition what will happen afterwards, the processes. So focus on the bill of material. I promised you that five minutes ago. Since may twelveth, the president, Joe Biden, has issued an executive order. If you want to read a little bit more about that, there is an excellent paper on TechCrunch of Ben Higgins that explains very well how that executive order works. Basically, any vendor has to publish a bill of material, of what is selling. It's very simple. So the typical use cases of the bill of material, it's used for automation, know what you have. It's used for compliance, for security. This is what we're doing and for understanding the complexity. So the bill of material has become an artifact, but it's an artifact that describes all the artifacts that are used or that has been generated to deploy the application. So just like in 75 Unix introduced Yak, which is yet another compiler. Compiler. For those who know that feature of Unix, I would say that the bill of material could be yet another artifact. Artifact, because it's a meta artifact describing other artifacts. If you are interested about that. By the way, in other conferences I have delivered a presentation which focus was the bill of material. With all these standards and the explanations of how to build a bill of material and how to use it in the software supply chain, we consider it as the single source of the software delivery proof. So the bit of material time ago described how we were building the software. Now it describes how we are releasing the software. It's just some kind of extension because time ago we had some sources, we had a build methodology and we produced some executables. Roughly it was done okay, now we have to deliver. So to go through a pipeline to do a certain number of tasks and to deliver the application in production and to see how it works. So it's just an extension. But since it's an extension, the content of the bill of material may change over time because I will analyze, I will scan, and the results of my scans of my analysis because of the audit trail are also part of my bill of material. So it's very important to understand that notion. It can change over time. So it's very important to make it immutable so that I keep trace of all the scans of all the different steps that were used to build my bill of material. So this is roughly the explanation and the presentation of what we did. So if I sum up in three sentences a security pipeline, which is the same for all the applications and all the technologies, an application pipeline that can change, obviously, but that will always call that control pipeline, that security pipeline. I say control because the security pipeline is not the only one. There are some other pipelines that are called still executing in parallel, still synchronizing with my main pipeline. But I am sure whatever my department, whatever my technology, whatever the application I'm releasing that if I got the okay to switch to handoff to production, it means that I am clear for the security aspects and for all the other control aspects that has been defined at the bank level. A couple of lessons learned. I made this slide, just one, to be very focused and to really understand what we've done, I would say, rather than some technical aspects that were very interesting. But one of the first lessons is implement. Do implement. Okay. DevOps processes, DevOps controls, automated bill of material. Just do it and do it now. Even if it's with a minimal set of tools, even if they are internal or external, even if it can look difficult to gather all that information, the return on investment is tremendous. It's fast and tremendous. So do it now. Do follow the state of the art company wide, because the competitors do. If you didn't do CI five years ago, the competitors were doing it. If you didn't do CD two years ago, the competitors were doing it. If you don't orchestrate your releases today, if you don't do devsecops today, if you don't look at value stream management today, the competitors do. And they will get an advantage if you do not do it before them. So do follow the state of the art in your company. And the third lesson, with the automation that we've implemented and the self embedded controls like the security ones I have described, it's very easy that the culture and the best practices will get disseminated within the organization. So at these, end of the day, what we have noticed in that project is that at the very beginning, users were some kind of trying, okay, they were not very keen to be chosen to try that new system because they didn't know what they were wanted. They were not sure of what to expect from that. They were afraid to have a constraint of going through security and so on. Now we have a queue, okay, very clearly we have a queue of people asking to be embedded into the system. And the only slowness that we have is from the ability of embedding new people, onboarding new people, and growing the infrastructure. But roughly, this is something that we managed to do. So I thank you very much for your attention. Please keep in touch. You have my contact details here on the screen. Do not hesitate to contact like me, and very happy to have been part of Confo v two these year. Goodbye and see you soon.