Thanks for joining my presentation, Secrets Revealed, for launching a successful DevSecOps program. In this talk, I'll be taking you on a journey of standing up this program in an enterprise setting from scratch, and then revealing the secrets along the way to my success of this program. At the bottom corner, you see a picture of me, and the right is my LinkedIn contact, if you'd like to LinkedIn with me. Here's the agenda for the talk. I'll go over the goals I had for the main parties involved in this new, program, the engagement strategy, how I got buy in from the, key stakeholders. First, the highlights, and I'll reveal the success patterns, and I'll close things out with how, SBOM will shape next steps in the maturity model of DevSecOps. When I first got this project, and scanning up this new program and the enterprise, I thought to myself, what does good look like? DevSecOps. What do I want this to be? I thought about, you know, the course I've been working nearly 30 years. When we did these type of things, all the places I've worked, what went well and what didn't go so well. And that's a lot of thinking about that. And I decided that things had to go a little bit differently if this was going to be successful. Obviously, I have to involve the DevSec developers, the security, and the operations or IT operations group. So I came up with this mission statement that DevSecOps for the company I was working at when I was doing this would be a successful marriage of IT operations and security enabling developers to deliver products for the business in a secure manner without any friction to their ways of working and while they're doing their other daily work. So first, the goals. I have this broken down by the three main players, developers, security, and of course IT operations. So the goals for the developers, again I alluded to this before, I wanted this to be as frictionless as possible for them to adopt. I use this mantra a lot, that secure coding is a form of code quality. We weren't introducing new, ways of working. It's just, let's try to focus on making the code, when it's delivered, have less defect. And also prevent rework from deployment, before deployment. And continue to deliver stable and secure products, not just for external customers, but also for the, the main company as well. at the time we were transitioning to this newfangled thing called the cloud and whatever we could do now to make sure that the patterns that we came up for pipelines and et cetera could easily transition to that platform and a place where I worked before what I said was analytics, there's always a focus on automation. What can we do now and in the future? Secondly, what goals do I have for my team? Well, I obviously wanted to decrease the application attack surface. And, a fun statement about that when we talk about decreasing the application attack surface, it's kind of, I kind of liken it to when people say, hey, I want to lose 10 pounds. Well, when people say that they want to lose 10 pounds, but they also mean I want to lose 10 pounds and keep 10 pounds off. So in terms of the application attack surface, I want to decrease my application attack surface and not have it grow for many other software app vulnerabilities as well. I want to mitigate the chances of any breaches or outages while we're emphasizing this new form of code quality with security in the foreground. And I used a blend of a, or a blend or approach of bend, but don't break for the security approach. So if we had to be a little bit less structured or, not as rigid with our security approach for certain items, we had to be flexible. I would challenge my team to think in terms of what detective, preventative, and corrective controls we put in place to mitigate any of those things that were going on from a software or pipeline perspective from a bending, but not only throwing the baby out with the bath water and not breaking, of course, always a focus on automate and closing out this piece, for it ops. I wanted their daily operations work not to be interrupted. I had an empowerment focus. I just didn't want to shove tools down their throat. I wanted them to have a say into it doesn't mean they had the final, call, but I did want their input. And also, frame this in terms of, upscale opportunity, not just from new tools or platforms, but also from a new way of working DevSecOps at that time was becoming very hot and you need to get on board with it. Right. And of course, automate. What can we focus on? Focus on automation now and the future. Moving on. So how did I get buy in and what was my strategy for engaging all these people? So I focused on the engineering management, getting alignment with them. the second piece is really a foundational piece I think anybody can do in their org. Education and outreach and then matrix teams. to be honest, I didn't know that was going to be such a secret sauce that I was going to rely on towards the end that really came through. So, so how did I get, engineering management alignment? So it was, this was at the beginning of the year and we had a focus, we're developing our, performance goals and objectives. I had some meetings with some of the VPs of software dev and engineering, et cetera. I wanted to make, security a form of code quality. That was a goal, that would be cascaded down to their direct reports and all the individual. So, you'll find out why, in a second, why it's important. Oh, I also did, address the budget barrier in terms of, If I'm asking these developers to operate in a different way to look for, for fixed security vulnerabilities or code vulnerabilities rather before they get into production, I need to give them some adequate training. So I talked to finance. I told them what I was doing. They knew about the DevSecOps program. I was able to secure X number of dollars for training just for the developers and other folks that would need it. I wanted to make sure it didn't come out of my team's training budget and not out of, engineering training budget. And then I talked about the education. I make sure they have the right, the right patterns to deploy. So what does this do? Well, the first ones got rid of, we don't have time to do this DevSecOps initiative. Well, now it's a priority because it's part of your goal and objective, and you'll be graded on it. I even went as far to provide sample SMART goals for the managers, such as, Hey, my team discovered X number of insecure, codes that they've written. And then they're able to, fix that before it was pushed to the prod. From a budget perspective, that's the, statement I would typically receive. We don't have money to do this new initiative. Well, I got training for you guys, and I'll go through the education here, but they didn't have to worry about that. And then this relieved that part. we don't know how to do this effectively and efficiently, and this is gonna slow down our work in delivering, not just secure code, but any type of code to our client. Thought about the barriers, how to address them to remove those obvious ones. Now from an education approach in the first year, I started off what was called the internal learning content. Every place I've worked there's been an internal learning management system and you can generate content for that. So essentially what I did was I went to OWASP, I got some short videos and content for them to go through. And in order for the developers and data scientists, et cetera, to attend the on site training class, which was the first thing I got for them, they had to take this OWASP. it was basically 45 minutes. You could do it during lunch, but it also told me who was going to have some skin in the game and also be more engaged when they went to the training class. So I worked with one vendor. It was on site, as I alluded to. It was two days, 16 hours. Eight hours a day, it was on site and they came, the instructors came here, to the office and I had about probably 40 to 50 developers, you know, engineering type, data engineers, data scientists. So it was good, from the content was solid. The vendor was great and this was a few years ago and some of the other stuff that I used down the road wasn't available. But. The, having developers always think like web app pen testers is not the easiest thing to do. They need to write code and they need to deliver it. So, you'll probably get one or two people that really gravitate to that model. But at the end of the day, you need to treat developers as developers. I've seen work and have them like look for these patterns. They should not, probably should not write their code from a security perspective. after the class, I started what I called a Lunch and Learn or Hack Club. Gave it a sexy name so people would attend. so basically during. The Hack Club sessions, I went through the OWASP top 10 to reinforce concepts that they learned during the class with the on site vendor, monthly. And that was well attended, and it was a good refresh, and like I said, I had not just, developers and data engineers and et cetera, but also a lot of IT people showed up to it. Now, the second year from an education approach, I learned about this, item we had called University Jumpstart Class. And what you would think, people fresh out of school or whatever. I wanted them to make sure they knew about the OWASP internal training path and make that part of their curriculum. I worked with another vendor, for a different type of, training called Modern Web App Pintesting, again. Still web app, still web app pen testing. Still asking developers to think like a pen tester and not just write secure code as you're going. The things to look at for. But this was a step up because the content was good. It was some more modern languages and frameworks they're used to, focusing on. And secondly, the, it was remote. And it was still 16 hours, but it was 4 hours a day. from 12 to four and for four days. And so that was an easier pill to swallow in terms of getting people to go, for moving that, Hey, I'm gonna miss two days of work versus, well, I'm only missing four hours of work. And really it's three cause they're doing part of it on their lunch hour. And so they're not so disengaged from their work day and you're starting at 12 and you're ending at four. So that was well received and I, I got a lot of good feedback in terms, not just from the individual contributors, but also from their management in terms of. Hey, this content was good, it was better, and I didn't have to use so much of my work day time. So, making things better, from a process improvement or curriculum improvement perspective. We continued the DevSecOps, or we started this thing called DevSecOps Office Hours Weekly, what it says, if you had questions from a security operations perspective, or, needed some help from, development, then. but intertwined with those three concepts, I had office hours. I had myself or some representative from my team there. I also had some from a team called platform that read all that ran all the I. T. ops, and so, you know, like build team city and cetera. And they could ask questions. Now, if it became something you really had to focus on, we asked him to put a ticket, but it was a quick way for people to get feedback and answers. We're continuing to push. This program, in the foreground and keep it positive. And then I found this other class, called Application Security Foundations class. This was exactly what I was looking. the content was how to write secure code and why. And I was able to get them a book from the author as well. They could have a print or a digital book. But more importantly, it really, it was written by a developer for developers how to write secure code and how to find insecure code and the best way, and the best ways to fix that where it was like libraries or different rammer stations, et cetera. This was met with very high success and it was all done asynchronously. It was not done live, so they could work at their own pace, but it had some skin in the game when they had to get it done by. And there was actually, there More classes too, and it's this one here called Secure Coding Principles. So once they got the AppSec, foundation is done, then they could go to the Secure Coding Principles class. also continue the Hack Club Lunch and Learns, but this time since it was year two, I passed it down to people in my team. So they could get some leadership, build, learn how to run those meetings. a lot of them are remote. And I made it part of their goals and objectives that they had to run, come up with content and lead one of the Hack Club Lunch and Learns. Now from the year three, round this out, I passed 100 percent off of the DevSecOps office hours to one of the people on my team. And I had some budget left over. I brought the author in from that class from the last one, application security. coding classes in the books. I had her in a fireside chat. It was virtual. it was about an hour and a half and I was able to have developers and data engineers kind of pepper her with questions in terms of, best practices and, things they've run into and things that she's seen in the field, it was well received. Plus I had a fake fire, you know, fireplace backgrounds that made it cool, especially since it was in August. So, moving on to matrix teams. So, came up with this concept of the star chamber. And the mission statement was to drive security throughout the dev process and the pipelines, or CICD process pipelines. By leveraging this group's diverse skills and perspectives to achieve, these outcomes for the business, but in a secure manner. So, there was this movie called The Star Chamber, and these people would get together and, make these decisions when they're unhappy with the law and kind of take action. Old movie from the 80s, I think. So, the group I developed and led wasn't that McLevanian, but we still did a lot of things behind the scenes in terms of pushing the program ahead. Thank you. Here you'll see the members of the start chamber engineering, development, developer services. I'll get to that in a second. SREs, cyber level engineers, platform secure or platform team. And of course, IT security. as I'll get to the, dev services a little bit later in the talk. the matrix teams, also started this group called the security champions program. And this is probably right now. This concept is pretty ubiquitous across bigger organizations. I've seen. With my clients, but this was probably three years ago, that was a new novel concept and not very well in practice yet much, but essentially these champions, was, push security concepts or advertise them throughout the respective dev workstreams. He also created this sassity, teams channel and get feedback quicker. You can see the type of items we have there and developer services. What I alluded to before. for listening. It was a small group of developers. This was, they already had this here. I had nothing to do with it. But I, I piggybacked off it. They had these patterns called paved roads. and then, you know, if you follow these paved roads, if you're a developer, you know, everything should be hunky dory and fine, and if you didn't, and you didn't work, then you, your code would not be blessed in production, etc. So, infuse the secure coding principles as part of their paved roads. That was a good win. and they also had office hours too, so I would join that as much as I could to hear what was going on from, from a coding perspective. I tried not to talk too much because it wasn't my meeting. I only really said things in terms of supporting what the leaders of the dev services were doing. and also I would do some shameless plugs for my, my own DevSecOps office hours and advertise, Hey, I've got more training coming. If you've not gone, talk to your manager and I'll get you in the class. So I've said a lot there. And what were the highlights of this program? It's broken down by year one and year two. So for the first year, I did decrease the application attack surface, by influencing the, revenue generating applications. and the dev programs that support those adhere to the SAS tooling, SAS tool compiler for the Build Breaker feature. So the results were 100 percent of the revenue generating applications were compliant by that time in July in year one. And then that was like the number one and number two RGAs, over 90 percent of those were compliant by November. How did I do that? I'll get to that in a second. we also increased code quality, helped deliver stable offerings to our customers and the parent company. And I used the SAST, the dashboard metrics to, drive those two components above. I'll get to that in the next slide. But I talked about automation and maturing processes. we also baked and a person on my team had a good idea to break the bake and an automatic exception process to be part of our ticketing system. where the sign offs were all done digitally by clicking a box. and then they were able to get a timed exception, you know, it wasn't a get out of jail free card, but if they had to get code out the door, they would do this process. And it also helped myself and my team know. Hey, do I have to increase my detective preventive corrective controls? Do I need to go talk to a different IT group networking to get a more aggressive firewall stance? Do I need to up my, WAF rules? Do I need to have a more aggressive, SEM logic where the logs are being sent to? So things of that nature. and then if people were saying, well, no, we can't do that, then I already had the data points to say, well, if we can't do that, then this code can't be pushed and we can't push this out to the, this feature out to the client. Everyone is aware. I had some money left over in the budget, and I had a third party vendor, validate my roadmap for the rest of the two years, and they said it was good. So, I talked about this a little bit before. This was a dashboard. This was just done in Grafana. what I like about it is, it's easy, it's simple, it's clean. If you're green, you're good. Yellow, not so good, and red was bad. And you could click into those dials and those other components on the left and right. And that would take, you to your devs, your, your respective dev stream to your dashboard of where you're, why you're failing this. You're not meeting that compiler breaker perspective. And what you could do is you could, click on that and it will take you to the, it will also take you to the tool and then there was comments in there from the tool from the, people that either fixed similar problems that you're looking for. And you could look and see how they did that. And if you couldn't figure out by looking at inside the tool, how they fix that, you had a, a person's name who fixed that, that similar problem you may have, you could reach out to that person and figure out, Hey, how'd you do this? So it worked out really well. there was also a, a big push from upper management in terms of having metrics and dashboards. Of course, I had a countdown timer, not a real one, but it's like, Hey, I'm gonna launch this dashboard. I'm going to give it to the folks, you know, the higher ups, in 90 days, nothing, 60 days. 30, when I got about 30 days, people were like, Oh, I better start efforting this. And once I sat down with them and went over just like, you know, If you fix this one thing, it's going to take care of 80 percent of your issues, and it's going to make your dial go from yellow to green. Like, oh, okay, that was helpful. Took it off their backlog, and then they put it on the front burner there. From a year two perspective, Things improved from a process perspective about, billbreaker compliance for all the devstream programs. They're operationalized, year two of Q1. I added security evaluation points into the CICD pipeline. I use the term evaluation points and not tollgates. I don't like, I don't know anybody who likes paying a toll just by default. It has a negative connotation. So I call it evaluation points. I talk about that in the word marketing and we did research a DASD solution. We tested it out, as recommended by our parent company. But at the time, the way things were going, it just wasn't the right time to introduce that, item into our DevSecOps, program. It would have caused some friction and the value that we would have gotten from the tool would have not been proportioned to what we needed at the time. So I rolled up and presented that to the folks and they understood. I also have this, in the tool there's this thing called Hotspot Champions and I empowered certain developers to have these review and fix the findings. They have like code smells and, they can fix it. Now, if they thought the way that was looking at something wasn't correct, they had the way to bypass that. So it wasn't flag. However, I would say there was very few people and the system that was the fast T system. I had all the logs going to the sim. So I knew when they would do that and I tested it and I make sure the rules fired as designed and I would run monthly reports and give it back to their leads or their managers. So they just knew that. Not like it was trust, but verify, but it was also showing like, Hey, your team is using this, you're using it. Well, it was also my way of like, I'm watching you and no, you have to do that. Right. In close to the end here. So I'll go over the, success patterns. And I have this broken down into two components, the technical and non non technical side. So from a non technical success patterns, It was partnering, listening, partnering, asking lots of questions, reoccurring meetings with all the engineering leadership. some monthly, some weekly, depending on their schedule and how far they're up in the chain. shout this from the rooftops, secure code is a form of code quality. And big, big, big focus on seamless experience delivered to developers while they're adopting this new process. The training and outreach worked very well. You know, we talked about this, the web app pen testing was okay. The hack club meetings were great. I'll mess means internal learning management system was better and the secure coding training. I did the end was the best approach, but if I went back in time, I would just focus on these 2 pieces and I would still do that. Hey, if you want to take the secure coding train class, I have that's not coming out anyone's budget. Not even mine or the engineering's. I got special money for it. You know, do the IMS training. OWASP training, attend the hack club meetings, and then you can come take those two secure coding training classes. And I, I just used this as much as I could and overused it all the time. You know, the form of code quality is just secure coding, right? So, part two of this, non technical pattern, the secret sauce, the matrix works, matrix working groups, the start chamber, security champions, the dev sec ops office hours, word marketing. So, like, that's it. Instead of using security toll gates, I assessment points, evaluation points. They have a lot less negative connotation, but they're still doing the same thing. Being the training dollars, from finance for a special, that the special project, I got it. The injury management didn't have to do anything, just had allowed the people to go. I also worked with, most forgot this. I, I also worked with certain developers and when I brought in training, I wanted their two cents to, to make sure they thought their voices were being heard. But also I had the final say, but. I wanted to make sure they thought, hey, are we covering the frameworks, the languages we need, are we missing anything? Get that sentiment. And speaking of sentiment, I did do some measuring of sentiment. how the program was perceived and how it was going. Not just from those SAS, like that dial matrix with the Grafana dashboard, but also just in general. sentiment, sentiment measuring is a big, big thing to do when you're measuring. If you. If the, if the perceived value of what your program is doing, not just from a technical perspective, but also from a non technical perspective, and obviously, if you get those two working in concert, it's easier to keep that process going and secure funding for other things. You need to keep it, going forward at the time. I was the director of security. So I'll give you these, from a leadership pro tip perspective. We had internal mechanisms to recognize people that did a very good job. And I always took this to recognize developers that were positive about jumping on the DevSecOps train. That were truly genuine about looking for vulnerabilities that were more security related and mentoring other people and learning how to fix those. also when I would write the write up there for that, I would enrich enrich the recognition early, nominations infusing the company values and mission statements where I worked for the C suite is very big on that, which I agree with. but if you infuse it with that, it paints. It has to be genuine, right? Like you can't just fake it, but you genuinely recognize a developer that is doing well as helping your security team either, directly or indirectly, and then you enrich it with the value statements and the mission statement from your company. one, the c that's gonna resonate better. It's actually gonna resonate a lot louder with the C-suite. And two, it's gonna show you as a leader that you're, you're listening and you're infusing those value and mission statements in there as well. And so it put, it puts you in a good light. You're nominating, you know, you're recognizing developer using the company values, mission statements, and so you're getting a good, Synergy, if you use that word, but synergy between, you know, leadership, upper leadership and individual contributor and middle management down, you know, to the developer level, I know places I've worked, you do your end of the year review, and then you would meet into a room and do some calibration, you know, I, I sat in and all, the whole organization's calibration sessions with an I. T. Not just, you know, My team and general IT, but also developers, you know, we're talking about people that, Hey, they did okay. They did great. They did really good. you know, one of the things I, I, said when we first started on the education journey of this, I don't know anybody that ever got a meat sum or a failing grade in terms of their end of the year review by saying, you know what, I fixed a bunch of insecure code before it was pushed. like, yeah, that's definitely worth, me, some, I mean, it's not right. So I also framed it like that. I highlight the things I think are important here. Moving on to the technical patterns. the static code analysis solution we pivoted to and went to, the very low cost point to get into and a very low entry point for the integration pieces adoption into the pipelines. That was good. it was purchased by my group, the security team, which people liked because they didn't have to worry about the licenses and it didn't affect their budget. There's a theme there. But the agreement was that I would buy it, but the platform team, which is part of ITOps, would run and administrate the tool. Which is good, because I wanted my team to deal with looking to do more security related work, not adminning. Again, Metrix, that DevSecOps program, we measured it with the, near real time dashboard, and just the ease, ability of clicking through that and getting to the tool, and finding out where you needed to fix your item. And that led to that self, Self feedback from an application perspective and getting those things remediated quicker. And a big focus again on that seamless experience delivered. example, leveraging our existing ITSM asset management application system. The automated and timed exceptions. And quick, and giving feedback quickly to the dev, or through the DevSecOps office hours. And the Teams channel and piggybacking off dev services office hours. I also like to say. There's an exception, get agile free card and everyone's risk tolerance is different. So just be mindful of that. But if it's typically within parameters and you know, if you did for like a week and you have some compensating controls that mitigate the risk, you know, it's good to get that done as soon as you can. But you can also run reports within the system to see, hey, how many people or what groups are requesting. The most exceptions, why? Is it a group? Is it a term? You can run metrics on that to your reporting, like is it a certain framework, language, etc. So next steps, SBOM and DevSecOps. So SBOM, what is it? Now this has been out for a while, but stands for Software Build Materials. And essentially it's a list of, I call it ingredients, components, libraries, and modules that are used in a software build. it was first started by, one of the first, our first people to use it was SPDX and CycloneDX. And they have different attributes and things that they, categorize that part on. you know, the goal would be automated, you know, put that into your pipeline, some process that you're able to break down the components and get a better visibility into. So this is driving by an executive order that came out in 2021. You should know that. And it's looking to, beef up and bolster the secure software supply chain and also deal with upstream customer expectations. Now with SBOM, you can, in theory, prevent issues before, it become a really big, problem you have to deal with down the road. You can detect vulnerabilities earlier and often and provide some transparency. It gives that insights to the engineering team. Like I said before, you can run some reports if you're capturing that data. Is it, are teams putting in more exceptions? Because they use a certain framework or is it a library they're using? Now you can get down to the nitty gritty and some components there, right? So how can this work in reality? Now I use these, these two CVs are a bit old, but they still work. And it's just a good example. These came out in October, I believe in 2023. and they deal with a lib curl, and, piece there. And then, you know, like, well, I have to find all this, all this, all these libraries and where are these, where are these located in my environment? So it's kind of like, where's Wado? Like he used to play, where's Wado? You'd find Wado, right? But in this case, you got to find his hat. Okay. finds classes, you got to find Waldo himself. So you'd be able to trace all that back. it's kind of like the cheat code for S bomb and closing things out. Here are the seven key takeaways. We'll identify remove adoption barriers like I did with the Making it a part of goals and objectives. Removing the budget barrier. very collaborative centric. Word marketing, I can't stress that enough. As long as the intent of your process or tool is doing the same thing, I would just soften the name of that. Don't use a toll gate, use a valuation point. the training and outreach, it shows that you're trying to help them, you're giving them the right content, you're involving them in the decision making process in terms of what you're Training to use and what vendors, but in the day you make sure you have the ultimate control over that and you're delivering the right content with their help. don't be afraid to advertise your program value. I would, I would tell it to my manager. If I would run into, one of my C suite, I would tell him if he heard anything about it and how it's going. and I also use that dashboard. obviously we had some quarterly updates as well that we used to do, but. But beyond that, I think it's always good to keep that going, you know, keep it in the foreground, always keep advertising your positives of a new initiative, brag about it, and then, you know, process improvement beyond what good looks like we talked about at the beginning of my talk and, you know, I like this number seven a lot. I don't think it gets enough love, but document and action your future. you know, I talked about automation at the beginning, you know, sometimes you didn't have the time, you didn't have the resource or the talent or the special skills to do some automation people list. I know it sounds corny, but that's why I used to do. I used to get what thing was called favorability, which was like some extra money at the end of fiscal year. they knew that I always had a list going in terms of like, Hey, Josh always ready to spend money. He hasn't always teed up, you know, if it was OpEx or CapEx, I could spend it on, you know, Consultants, or I can maybe buy a tool or extend some license I needed, so, but anything I could do to automate, that was always one more thing, because one, it was one of our goals, but obviously it's just a good thing to do, but if I was able to automate some stuff with my team's help or anybody else, that was another thing I just threw out there, or advertised in, the positive aspects of the program getting more mature, and these are the ones I think are really important. So that concludes my talk. Thank you. I hope you enjoyed it, and if you'd like to reach out to me, you can, find me on LinkedIn. It's my contact. Thank you very much.