What is DevOps? DevOps, for me is more a culture, an approach of managing your projects, and so on. But I think once SRE came up in the picture, then we had actually toolings keep track on DevOps approach. And then now today, they are introducing a new role, which is being out there everywhere on the Internet, on the blog, on videos, and everywhere on platform engineering is the hot stuff. It's just that I think we were asking a lot of things to people, I think new roles pops up to resolve a lot of pressure put on the shoulders of engineers. I think platform engineering is a great thing, but I wonder if people fully understand what is behind platform engineering, what is not behind the platform engineering. And sometimes it's a bit confusing. I mean, I would argue that DevOps was never born, and because of that, it's not dead. And I don't want to reveal my age here, but if you go many years back and look at software engineers of 1980s, they were on the full spectrum of what it takes to write a code, get it out there, make sure it's run properly, and look after it. And then there was a period that we started to have a lot of different technologies, explosion of programming languages and technologies, and then specialist front end or middleware. And then gradually platform evolved and we started to see that, okay, we're creating silos. And then DevOps came about to break these silos and connect engineers back again. And now it's taking different guys like sres or platform engineering to perhaps do the same thing. What sres do, they go. At least one way of implementing is they embedded in the teams to make sure that the team understand your piece of code is not running on ether. What are the other components of it? I want to add to that, because I feel like part of why DevOps came to be is because software projects kept failing. Like with waterfall, something like only 30% of the projects would succeed. Obviously, all of mine went perfectly. Of course they were right. But lots of my friends projects failed. And part of the idea was when we switched to agile, it was like, what if we asked for feedback instead of building a thing in a silence for this really long time? And then with DevOps, it's like, what if we spoke to ops? What if we didn't, like you said, have the two silos? But also, what if we got feedback all the time? What if we tested all the time? What if we integrated constantly, your piece and my piece, so that in ten weeks from now, when we met and we tried to integrate it and it didn't work. And then we're both like, it's your fault. We've already tested that all the time, in my opinion, is like a lot of the magic, because I started seeing projects go just better and people getting along. and like you said, henrik, like culture change, it required so much change in the way we think and work. So I come from the performance engineering background, and when we were in waterfall, it was like we were doing things, and then we have to go to production, and then we were doing some optimizations, and the ops people were completely different. And the way you deploy, the way you were doing things, it was a consistent fight, or it was like going through the journey of explaining everything to another team. And that was so time consuming. I think the fact that now we merge the ops part of the development process makes the things much more easier. And I think to be fully DevOps, I think you need a platform that allow you to deploy from a code perspective. and store the assets of your, for example, kubernetes is, I think, one of the key driver to enable the DevOps approach. But if you're still in the old fashioned technology, it's more difficult to implement those automated processes to deploy things. I mean, at least that's my perspective. I think it really depends on what you have internally to fully apply this approach. Sometimes it's very difficult to put a triangle in a square or a square and triangle, because sometimes you force things. We want to be DevOps, 100% DevOps. But yeah, it doesn't fit. It doesn't fit, but let's change the shape instead. Yeah, I think with the emerging of cloud, cloud native technology, I think now it's much more easier to get started and implement that. Yeah, I think part of it too is the evolution from waterfall to agile. Also mimic the evolution of shipping software on a CDrom to it always being online. And you cannot export all of your performance aNymore by shipping a cDROM to a customer aNd expecting them to install it, and then pay your consulting arm $10,000 a week or $10,000 a day sometimes to install it and manage it for them. So the loop is closed, right. You're getting more feedback from the users, and that means the developers have to be closer to production than they were when they were just shipping things off and putting cds in the mail and not caring about stuff anymore. So the whole evolution kind of mimics not only the software development lifecycle, but also the way you manage your teams and the way that your organization has to function because now the loop is a whole lot closer than it had been when you were shipping cds. So I think that evolution is still missing. Some people, I think, feel like there's still some developers out there who really haven't fully engaged with the idea that you're there to produce a thing that the customer wants and needs and is going to pay you for in most cases. And you want to get that feedback and incorporate that super quickly to get things better for the customer. That's a very good point because it kind of stretches in two way role of software engineers. I mean, DevOps is pulling it closer to operations high runs, but the other side that it should stretch is to understand the customer better. And I don't know, is there a name for customer DevOps? Is there a name for stretching that way? I think it's the so approach measuring how good your software is. And before that we were using, I don't know, centic monitoring, or we'll use a monitoring to react on something that was happening. And now we try to be aware before the actual user starts to see it. So I think we are getting better. We're getting better for sure. I remember being a software developer and wanting to show my progress to my client as I was working, and this was like 2012, this was not that long ago. And my boss saying like, no, they will wait until it's done. I'm like, no, I need to check with them to make sure I'm building what they want. And he's like, well, they're just going to change their mind all the time. And I'm like, here's the thing, buddy. I don't want to have to rebuild this in six months because I didn't build what was in their brain, right? And if I can show them like, oh, it's going to work like this, does that make sense? And draw pictures and do mockups and whatever. I am bad at making things pretty. I know I have this weakness, but they got the picture right, like a menu here, drag this over there. I'll make it look pretty later. And by I, I mean another person that's good at that. But do you know what I mean? We'll get this across as a bit of context. I now work in security. I now do application security. So I talk about devsecops. So I'm that person that thinks we need to bring security up a lot because it's not always happening. So I'm going to be injecting that into our conversation a bit because I cannot help myself I was wondering if the rest of you could give a bit of background of where you're coming from. Because I was a software engineer and then I switched to security and I briefly did cloud security because I couldn't make up my mind, but where each one of you are coming from before we start talking about the trends we're seeing. Yeah, Henrik, kick us come from. I used to be a software developer. I moved to performance engineering, where I spent most of my career. And then recently I moved from performance engineering to observability, which is a requirement for performance engineering. And I'm more touching, more cloud native topics now today than I did before because, yeah, you touch base technology that your customers are using and you don't explore necessary technology every day, depending on the time that you have. But yeah, performance and observability is my key topic. Can I ask, what is the difference between performance and observability for people that don't know? Like me, when you do performance engineering, you are driving the approach from the beginning, from the dev stage to production. What do we need to measure and what do we need to do to make sure that our system is performant? So performance could be. Many people think about response times naturally, and the fact that it's available and so on. But now with the fact that we move to the cloud now, there are also the aspect of energy consumptions of our data centers, the actual costs that we will basically been paying if we deploy this application into production. So the performance is not limited to response times anymore, it's larger. And to be able to measure, you need to be able to take decision, you need to measure it. This is where observability come into picture. We used to call it monitoring back in the old days. It's been renamed because we were mainly relying a few years back in metrics. That was the main thing. People were looking at dashboards, graphs, and we all had logs. And with the opentemetry project that emerged, distributed tracing has become very popular. And now the approach of obliberate is to take not only metrics into your decisions, you're taking all those various signals. So metric traces, logs, of course, profiling, if you do profiling, and then with all those signals contextualized, you can understand what is happening and you're more efficient when you need to troubleshoot. So at the end, when, if you don't want to do performance engineering properly, you rexed at least to have a proper observability stack in your environments. Otherwise it will be difficult to make decisions that's awesome. Thanks for humoring me. Who's next? Yeah, I'll go next. So yeah, I started out as a Linux systems administrator back when people thought that was foolish. And really, if you wanted to do systems should hamed done Solaris or Hpux and that panned out walls. Now I do community Devrel kind of things. So I currently work at Pagerduty. So we're working with incident response and helping people, figuring that whole process out and post mortems. And along the way I've done a whole lot of stuff DevOps wise as far as helping people modernize all of their practices. We do still run into companies that are doing agile fall. We'll say they get stuck along the way and are hindering their progress forward into modern practice with whether it's politics or fear or whatever the problem is, they just kind of get hung up in places. We talk about that too. I guess it's my turn. My story is a little bit complex. My background is civil engineering. So I graduated as a civil engineer. But I realized I just love programming. So after studying four years of civil engineering, I moved and studied software engineering and then worked as a software engineer. After a couple of times mistakenly deleting an whole Oracle database, I found a job as support team lead. I don't know why I got that job. Probably they didn't ask me enough questions in a financial trading company. And then now this is around 2013, I set up a team. I think that was the first time we invented this word of reliability. So came into space of there was DevOps going around at the time, then reliability engineering, and became very passionate and interested in incidents why system fails. and became big fan of locks of John Olsbo and Dr. Richard Cook, if you know them. And my journey as head of reliability engineering ended two years ago when I became a founder. So I left my job to set up a company called Uptime Labs, which our work is very much on. Incident response. and creating simulations and drills and getting better at incidents. One reason I did it because I was really missing coding. I said, okay, if I build my own company, I can do coding. That's not true. I love it for many other reasons, but that didn't work out. Oh my God, it's so true. I founded my own company too. And you end up doing management meetings and marketing and partnerships and it's like, when do I get to do the cool part again? That's hard. That part of me is still alive. One day I'm going to get back to coding, but they just don't know where. Well if you get acquired, you could do that. That's what I did, I got acquired. Now I get to kind of go back to do some of the stuff I wanted to do. Someone else has to be the boss. Yeah, I'm not in charge anymore. Hands off, hands off. You got it. So what types of trends are we seeing with DevOps? Because things are changing. Right? Does anyone want to throw some thoughts out there? I think what would be interesting is to why did we brought up platform engineering initially when this role came up in the picture, when I real reading that blog and they were saying oh, we're going to make an ops role back. And I said what? I thought that we were going to put the ops back in the same team. And then no, they kind of create a role separate. And then I said okay, so this is interesting. And once you go further in your topic, you realize that, yeah, if I ask my DevOps team to do their pipeline, their automation, but they need to deploy in kubernetes, so they have to manage also the knowledge in kubernetes. And then in kubernetes you have probably some security aspects, some network routing components to include and then you're asking so much expertise to everyone but nobody can keep up on this. So that's why I think it was really beneficial to put like a team that will be central, that build that apps catalog so then people can just pick up what they need to basically have the right assets for their applications and then focus on the things that they need to deliver that will bring value to the business. I think it's a new role that walls make the DevOps culture even more stronger in the organizations. I think that was a missing component and I think I'm very happy to see it. And I'm really eager to see if most of the organization implementing platform engineering, not just renaming the role because it's the first reaction. So I'm going to rename my DevOps whatever engineer into platform engineering. Now I'm more thinking of since we started that practice, are we more efficient in the way we deliver our software in production? I walls be very eager to hear about that because this role is quite new, it's less than a year old. But I think if you do it well, I'm pretty sure that it's going to bring a lot of value for the organizations, at least for making the automation more stronger, making sure that the observability piece that is required later on for the sres, the performance engineers or any other aspects is already considered very early when they build their solution. So I think, yeah, it's like the check marks required to make it successful will be managed by this team. So I think that hope that across the fingers, that is going to be valuable for 2024. Yeah, I think the interesting thing about product engineering is that it looked by platform engineering is that it looks like an internal product, like you are building your internal components to help your application developers utilize all of the products that you use to build your software. And part of that is your golden path or your paved roads or however you're putting that together. And that is very much a product building kind of mindset with your customers being your internal developers. And that's going to be some input from your SRE team. What should we monitor, what should the performance look like? What are we consuming for that? Plus all your application developers, what's the pipeline? What are you doing for integration? Testing what's the other components that you need and putting all that stuff together to facilitate everyone else's work. And I think that's a kind of role we haven't really seen for a while. Like the pendulum swings. We've gotten to a point where there's so much complexity that someone needs to go deep on this stuff, and that's not going to be your application DevOps. You want them going deep on your application languages, not on all the other SaaS components or whatever else you're doing. And so we need more expertise into some of these pieces for folks to continue to be successful. From a security perspective, I think that platform engineering is super exciting because, hamed, you probably are thinking the same thing when you're responding to a security incident. So first of all, you don't know till after unless you have some sort of observability going on, right? You get told after you're looking at everything after the disaster has happened as opposed to being alerted. It's happening right now. And that's super exciting. Like so many companies I've worked at, they're like, hey, can you go investigate this? I'm like, you have no logs. What am I investigating? You've literally made no evidence for me, right? And I'm still finding that still on the regular, they're like, oh, logging is really expensive, or monitoring is too expensive. So we just turned all of that off and I'm like, it's true. And then a lot of security tools don't work properly if you don't have those turned on, how are we supposed to detect bad stuff's happening. And so now that we have these platform engineers, we could talk with them and work with them and try to figure out like, hey, this looks like a security incident. Maybe if you see stuff like this, you could call us and we could come and check it out with you. Because if you can catch something as it's happening, you can save a lot of money if you can find it very quickly and close it off. I hamed done security incidents where I'm like, this cost more than my house, this mistake, right? And then if it goes on and on, it's just like every single day, worse and worse and worse. I worked with a company once and they almost, if you do business online and you don't have access to credit cards, your business is sort of toast. Most customers aren't willing to just do PayPal or cryptocurrencies. They want to use their visa or their Mastercard. And they had screwed things up so badly they almost got no more credit card. And so it's really serious, right? And so I'm pretty excited about this change and this focus on observability. I know that it's really good for all sorts of things, but the security people are having a bit of a party. Just so you know, from an observer perspective, we think that. I'm not saying that the expertise on security is part of observability, but the data that comes out of security toolings is clearly an angle for observability. Being able to, if you have a vulnerability alert, being able to figure out what are the user journey or what are the path the user journey in the application that are impacted by this issue. To then make a patch quickly and dirty so then people can buy is crucial because you say, oh, we are vulnerable, but where and who is impacted? And if you start asking those questions, then you are screwed for sure. So what is the boundaries of platform engineer? Is it, sometimes I come across it and include some dev toolings in some organization. I see that, oh, there's a platform engineering team and there's a dev tooling team or dev engineering or developer experience engineering. That's another one that I heard recently. Devex for me is if you have a platform like kubernetes, you will set up all the best practices to first utilize in a more efficient way. The resources that you may consume in your cluster provide all the default toolings that will make their development successful. And if they want to use Argo or flux or whatever to deploy, they can pick and choose in the app catalog, deploy it, and maybe provide like a pipeline template, because sometimes when you move on to another pipeline tooling, then you have to learn it. And people moving from Jenkins then to Argo say oh, it's completely different. And then people have to get onboarding into those new tooling. So having some templates so people can get started easily, easier. Yeah, I think it's also something and also not from an architecture perspective, but more. The platform engineer will have to check that everyone is respecting our best practices in terms of, I don't know, we have to set resource allocations values, otherwise your workload will consume forever. So yeah, just putting the standards. So yeah, the customers are the team and not the users of the applications. But I had a question on my mind and say, as a platform engineer, I'm pretty much disconnected from the value that my organization is providing to the market. So is there a way to link or measure which project are more important to the other based on revenue or based on, I don't know, on critical applications. So then if he needs to react to a request, you will make the priority based on which app is more important compared to the other. I don't know. There is a lot of questions related to platform engineers still out there, but I think it's a good sign. Our industry, IT industry, I think we are pretty young comparing to established industries like aviation or civil engineering. Decades and decades, and probably some of them stretches to 100 years. They evolved, learned the rules, became very clear, but it I feel still we are discovering what rules we need and how complex these systems should be. Sometimes I think like are we making things too complex to just deploy a hello word Javascript? With using these new frameworks, you end up downloading 50 megabytes, running five tools, gradle or whatever to compile it, and then hold the AWS stuff to ship it out, and that used to be like notepad and HTML command. And I'm not sure if it's like we're going too much complex for the same result or is a really good reason. That's a question I hamed love to hear your perspectives. I think you make a really interesting point, but I think the it industry has learned from the car manufacturing industry. So we shifted the agile and other processes based on how they were doing things. So I think if they're doing better, maybe in ten years we'll be as good as they are. I think that the apps we're building are totally different than we used to. I remember making an app that was for twelve people being like this is ridiculous. But I would make apps for 2000 people, for 3000 people. And then I went and worked at Microsoft and we're working on stuff that millions of people use. And when there's the tiniest memory leak in something, it's catastrophic. Like the smallest error, when you're scaling like that, it becomes ginormous. And so I also think that the complexity is in what users are expecting these tools to do. Now, like we were talking earlier, before you got here, hamed, about how I was on a flight that got canceled, I couldn't go into the app and cancel the ticket. I went and just bought a ticket from some other airline. I was like, listen, you've delayed me three days, you want to delay me two more? I'm not moving to London, even though it's a lovely city, I need to go home. And they just kept canceling it every day and I was getting pretty nervous. So I was like, I'm going to spend some money today, but I still can't cancel the ticket. And they're still rescheduling me. I don't know what they think they're doing, but the fact that I can't even go on the Internet and cancel, I have to wait on hold for like 8 hours. I'm like, this is unacceptable. But think about ten years ago, think about 20 years ago, you would never be like, I can't believe I can't press some buttons and cancel a flight. Now you can compare a zillion flights and you can decide a zillion other things, right? You could never do that before. So what the expectation is of the customer, I feel has completely changed. That's very interesting. So it's a market pool that demands complexity. I think so. I also think that over this time, so when I started writing code like in the weren't hackers all over malicious actors trying to get into things all the time? And now as soon as you put an app on the Internet, someone's scanning it, someone is trying to get in there immediately. And that's not a thing that when I went to college in the hamed to deal with, right. And so software developers are not only facing customers that are very sophisticated, that have advanced needs and wants and expectations, they also have attackers that are just like all over them. That's not stuff. When I was a dev, I remember I was just starting to have scans done before I switched to becoming a pen tester where I seemed like a genius because I had a little scanner that could just check the things, right. And things have just really catapulted in this new way. This is my opinion. What does everyone else think? Yeah, I think we're kind of suffering from our own success, like the ubiquity of software in lives and generally in the culture and all those things. And someone wants to sell you a smart washing machine. and I'm like, I don't really need a smart washing machine. I need it to wash my clothes, and that's great. I don't need anything else out of that. But incorporating all of that sort of digitization into all your regular life has really become something that, like you say, we didn't learn that in a computer science degree, and you kind of still don't. Mandi, it's going to look a lot different, I think, even going forward, as things continue to go. And it's not like we're intentionally injecting complexity. It's that things are naturally complex as you continue to add components to them. And when we were dealing with server workstation kind of conversations, that was one thing. And now we've got distributed systems and they're geographically dispersed, and there's lots of codependencies and other things that live in other places. And yeah, you have a lot more robust and sophisticated software, but at the same time, your environment that it lives in, the topology that it lives in, is in some ways much more fragile and sometimes very brittle. Even the complexity of the system is based also because everything is based on software today. A few years back, it was mainly apps or web apps, maybe, and now even the car. The car has software, has tons of software. It's not even a traditional car where you just can plug things and repair things. You need a computer to understand and diagnose the problem. So that shows where are we heading. Everything is based on software. So everyone is afraid of having the software dying and then being dead and then generating a lot of consequences. I have a customer which is one of the leader in the transport containers on the sea. And it's even impressive. So they have, every single container that is in the ship has a small IoT device. So they are pinging, depending on the container, what it is inside. You can also measure the temperature, things like this to figure out even when it's been delivered on the destination, if the transport was done well, and if the product has not been impacted or infected, whatever. And then even those company, now they are building even the networking. So then when they are in the middle of the sea, those containers is still able to communicate back to the main data center. So, I mean, in the 90s, we never thought about that. It's just that now we are overflowed by data and we love data. And just to see the explosion of AI implementations, everyone wants to take the data, make some great features about knowing our customers, suggesting great things to their customers, and behind you have data and so you have to process that. So even more bigger software, those software that process data is usually super expensive. And yeah, I think it's not only about the we are touching more people because we can operate from a small country, do a business from a small island, and you can basically deliver to all the people on earth, which is amazing. We never thought about that in sure, it's really interesting that the software and computer systems just getting into every aspect of our life and that I think puts a huge responsibility on it engineers, anyone in this industry. And I think probably we need to wake up and accept that our industry is now a safety critical industry, because system failure can have impacts far worse than plane falling off the sky, for instance, payments, transport, all aspect of it is dependent on it. And I was looking into this topic because of my interest in incidents and how organizations look into incident response, and I realized that software is so important, we are it. I would say it's already safety critical industry, but in terms of regulations and standards and practices, we are far behind other safety critical industries. If you look at aviation, there are specifications how to build a reliable plane, how to test it, how every component needs to, what regulations they need to comply with in it. That reliability is pretty much kind of relying that, you know, that platform engineer that I hired, he hopefully have read released it book by Michael Niegar, and knows how to build a system that doesn't fail and have resilience. So I think that's the next big thing for our industry to wake up to the fact that it's a safety critical industry and take steps toward that. And I'm sure regulations will catch up. I see already in financial services in Europe, there's a regulation called Dora Digital Operational Resiliency act, which is starting to better define how IT systems and financial services needs to be resilient. So I think that will be a big pattern. I don't know how from platform engineers we got to this point, but you made an interesting point to say that I'm very interested in the green it topic. I'm following that aspect as of now, as of today, just the software that we have in data centers, and in the cloud is 3% of the carbon footprint. But the problem is that we are adding more and more software, more and more data every month, every day is added. And if we don't do nothing, it would just keep this growing. The way we utilize resources, we're going to reach out. I think they say more than 16% of carbon footprint usage globally will be data centers, and just the airline industry is 10%, so which means that data center will be bigger impact on earth than just flying overseas, which is crazy. So I think it's a topic that people need to understand and try to build software in a smart way. And one topic I want to also brought up, you mentioned why are we not respecting all those processes? And I think it's pretty much related to the fact that technology is moving so fast in it world. There's always a new framework out there, there's always a new things to learn out, learn from. And there's a lot of open source project, because open source is innovation today, and keeping track and being able to know everything today for normal engineers is impossible. So it's a never ending learning job working in it, for sure. Earlier, when you were talking about all the data, we're collecting a thing that I have seen. So on the security side, and privacy specifically, we talk about data minimization. So, like a lot of marketing companies or marketing teams, they collect everything. Like, every time you take a breath, they would like to collect that, but a lot of them are. One, they're not using it. Two, it's none of their business. And so a thing within the security and especially the privacy community is like, why don't you not collect it unless you actually need it. If you have a data breach but you haven't collected that data, then someone can't steal. So I saw a talk by Troy Hunt last week, and he gets sent these giant data breaches for his site. Have I been pwned? And he's like, yeah, I never keep the. I I validate it. I make sure it's cool. He's like, I erase that. Do you want to know why? Because no one can steal it from me if I'm not posting it. He's like, I make sure this. And then I just have a bunch of email addresses and which breaches they were in, and then no one can steal it. And if more of us design our systems where we respect our users privacy, despite the marketing team's protest, we only collect data that we're actually going to use. Because sometimes it's just like, well, let's just collect everything, then maybe we'll need it, and maybe we won't. And if we're trying to reduce data center sprawl and the amount of energy we're using, this is a way where we could have better security, have better privacy, and maybe save the environment through data minimization. It's a thing I'm kind of excited about recently. But reducing the data is a big challenge, to be honest, because with the growth of everyone wants to see systems and observe them. So we're going to increase the data anyway. We are running in two circles. It's true you're in opposition to the ability to use AI then across all of it as well. That's computationally expensive and therefore environmentally damaging, as well as requiring large data sets to train and maintain those models. So there's a whole other discussion to be had there. What that looks like in the future. I walls wondering when AI will come up in this conversation. Who has been asked 400 times, what do you predict for the future of AI? Constantly, like every single event. What do you think of AI? I'm like, mostly. It's annoying right now, right? Mostly going to get good. Or the other question is, are we going to lose our job because of AI? Ask Chat GPT to write you a program. Yeah, Chat GPT. Rewrite my resume for me so I can find. We were talking about how software is getting complex for a good reason, and how big it's getting. We talked about the green it that henrik you brought as walls. The other factor is, can we really separate working software from people? And by that, what I mean, if the software is in production for how long? We can dare not to get any engineer close to that software and still be confident that it's running as expected without any operations person. And I do ask these questions in conversations, and I haven't heard anyone say in more than three weeks we can trust that software going on. Which then leds me into then, when we talk about software and systems, really, people are part of that and when we need to see it, but would be interesting to get room's thoughts on that. I would say that if you do like remediations, I will never have a system do automatic remediation, reply things directly without a human action. I would prefer to have an automated process that suggests a commit or suggests a change, and then a human looking at it and say, yes, that makes sense, so let's put it on. But I will never hamed some programs that says, oh, the memory is here, I'm going to kill this, because I think we are not mature enough to make these automated decisions without measuring the actual impact on the complex environment. So not even AI can do that. I mean, AI can, AI is just that, oh, I have this, I need that data here, here. And then you can basically suggest a change. But I will never put AI doing the actual decision and say, let's do that change now. I mean, maybe in the future, I don't know, but for now, because I like the fact with a system like GitHub, GitLab whatever, or BitBucket to keep track on the changes and know what happened. And like you mentioned before, and, being able to do a postmodern mortal analysis, you have a problem. Then if you have those changes marked somewhere, then you can understand and take actions. And if a system is doing things without any control, then I hamed no idea how we can understand what's going. So ops people's jobs are not threatened by AI, then that's good news for just, I think, ops people, because we're dealing with so much cloud environments, a lot of workload, a lot of data from different angles. So being flooded by data is very complicated. Having a system that guides you to understand what is happening and what type of action you should take, I think it makes sense. But I will definitely say that, no, I can't imagine any company yet being comfortable enough with the current models, the way they're built, to be able to take all of their data about their environments, about all the things that they have deployed, about how all things are connected. Even if they have that, which a lot of folks don't have all that data, there's still a lot of it that's buried in brains and not actually documented. Like you have to train that AI model on your system to be able to get good predictions back out of it. And I don't think a lot of these large enterprises are in a place that they're going to do that yet. There's a lot of maturation that would need to happen in the models to trust your data. You don't want everyone out there to know exactly what versions of every piece of software you're running, because that opens up your attack vectors to everyone who can read it. You don't want that out there. So training these models is very sophisticated, and you have to have all that data available to them. You don't have all that data available to human beings right now, let alone being able to ship it into a format that an AI model is going to understand. And I don't see that happening soon. Right now it's too expensive, it's much cheaper to deploy humans to figure that stuff out than it is to pay for the compute and storage and training of those models to get that. I don't think we're anywhere near close to having deploy an AI bot into your environment and trust that it's going to be able to figure everything out and suggest solutions for you. What about if the system where the AI rely on just goes down? So he's going to predict, I don't know what would be scary. Yeah, really. When copilot came out by GitHub, it was trained on all those open source projects that have no security team, that have no money for a pen tester, that have no money for a code review, or a security expert to come and look at all those things. And so almost all the code it would suggest was like, oh God, don't put that into production. And so Microsoft came out with security copilot, which is still, they think maybe next year it'll be ga and it's still in beta. And I saw a talk and I interviewed this lady from Microsoft last week and she said basically like, please use all the old controls, like, do not depend on us. She said it nicer than that, but she's like, it's not ready and everyone else is just lying and at least we're telling the truth. But I'm seeing some startups come out. There's one startup that came out and they created a lot of cool stuff. It's founders from contrast security, which created a bunch of security observability tools and runtime security tool, like very interesting, innovative stuff. And they've come out with an auto remediation tool and obviously there's AI and, everything. But I guess the idea is like, we've seen this bug four zillion times and we know you fix it like this. So we're going to make a pull request and tell you we think this code will fix it. Do you want to check it out? Because I don't know how many of you have received a report from a security tester, or more importantly one of those automated tools and they're like, here's 40,000 things wrong, get going. And you just like, I quit. I can't even, right. And so if you could instead receive a thing that says, okay, so we found these things. These are like the twelve things we think are actually of important, and we have a potential fix for eight. I feel like that might make developers not hate the security people so much, hopefully anymore. I'm pretty excited about this idea of am I going to just press a button and release the prod. With no testing. No. But I'm certainly cool with looking at it, seriously considering it, running all of the integration tests and functionality tests and unit tests, see if it works. And then it's like, okay, so maybe we should just do it, because there's a lot of little pieces, like you say, like, little pieces of functionality, like, oh, you left this port open, or your security isn't set to the right key strength on this particular account, or whatever, those kinds of really specific things. I think there's a lot of potential there to your copilot kind of thing will do for you. Like write me a library. Yeah. Fix my shaw problems or whatever I've included here that aren't right. Yeah, right. Oh, my God. Reminds me. What walls. That tool fortify the security. Yes. That's the static analysis tool. Yeah. It was like my nightmare, seeing the 50 warnings every morning. How many of them. I know I scanned an app once with one of the old static analysis tools, like a first generation, like fortify, but it wasn't them. And it said it had 43,000 vulnerabilities. And I was like, I'm going to go home now. One web app. But now security industry is moving towards next generation, where it does pattern matching and more accurate ways of finding things. So there's just way more true positives, and fewer, less false positives. It's still not perfect, but the accuracy is. When I started an app tech, I was like, throw those in the garbage. Why are we even using them? It's so useless. But now I'm like, oh, I like it now. It goes faster. It gives me results that actually, I was like, oh, that's terrifying. Thanks for pointing that out to me. Versus I hamed to dig for, like, 300 pages to find a thing that matters. So we're getting better, but we're not perfect. But I guess if any industry person tells you that, they're probably also got some magic beans they need you to buy. Any other comments on AI and machine learning besides I hope it gets better? Well, there is one small point. AI, you touched on it when this new copilot, they look at existing open source codes and replicate that. There's another angle to it, which is currently in the IT industry. More than 70% of the codes are written by male people. Okay. It's growing, but we have smaller participation from women in coding. Now, if AI learns how to code from the code that written by a bunch of guys, we'll replicate that code. And we may say that, okay, there's no difference the way women code and men code. And the reason that question came up for me, I was looking at, because of my job, going through labs of different incident logs, people's conversation, and we noticed actually women communicate differently in incidents comparing to men, the way they talk, what gets them engaged, what it's not. And we're contemplating that if there's an AI tool that can get engaged in the incident response and it looks at all these chat logs walls, be more like a male participant than female participant. And then the question came up, oh, is that also true for coding? Does Java code that a female software engineer writes looks different to what a male software engineers write? Can we answer that question? I don't know. But there's another aspect of AI, which is the gender bias, which it can project if it only looks at past, given how it's distributed in participation of the gender in it, I wonder if that's cultural too. So this is a weird story, but the first time I pen tested an API, so I'm canadian, that is my accent, you probably heard. And the responses from the API were super polite, just like the stereotype, right? And it was like, oh no, I'm sorry, that's bad input. This is how you want to form. And so it kept helping me until I made a successful attack. And so then I had to write them and be like, you're not allowed to. So the only person that's going to see an error message from an API is either like a tester, the developer, or a malicious actor. And so you're helping the malicious actor attack you. It should know, invalid, know, error number, whatever. Call if you need more details, right. And then you can validate who it is. And I had to reassure them that was okay. And then I was working with a team from India recently, and I was explaining to them, it's okay, you're not being impolite in certain cultures. They're concerned about being impolite with the way they design their software. And I haven't had to assure american teams, they're like, obviously it's fine, right? But it's forced into certain cultures to act a certain way the same, I don't know if you've ever read the book drift into failure, but they talk about how in Korea there's certain cultures that are different than in other countries, and the copilot just doesn't question the pilot. And so when some things were happening. Yeah, exactly. And so it's also with AI. I wonder if the way things are coded, designed, responded to, et cetera will be based on different cultures. And with women it's a lot cultural rather than our actual anatomy that's making us act differently. It's that we're expected to act a certain way. And if we don't act that way society gets all upset with us and tells us we're bossy or whatever, right? And so this culture thing, I'd be very curious to see what AI makes of code from different countries, different classes of people depending upon the place that you live, et cetera. I'd love to see the difference between indian programmers, canadian programmers and maybe south american programmers. That's super interesting because part of the thing with the airlines walls that certain spoken languages have different kinds of deferential speak. And the case studies there are super interesting. And there too if you've got english native speakers as your programmers versus non native english speakers as your programmers. This sounds like somebody's phd project right here. I don't know if anyone out there listening is looking for a thesis, but I'd like to read it. I think it'd be super interesting. Yeah, definitely. The book seems very interesting. It was a very interesting book about how many tiny little things where we drift just slightly off course years at a time, then becomes a major failure. And I've definitely seen this with large organizations. I don't have any answer for the coding side, but if you're interested to see how men and women respond in incident bridges, I have some interesting data. Happy to share. It was really fascinating for me to see how they play differently. That's kind of warm up. I won't open that topic here this walls, but it's very interesting. That's where I think open source projects are great because it collaborates developers technically that are sitting from all around the globe. So it's going to be a rexed of commits from different countries. But again, there is always the maintainer who will approve it. So there's always one guy sitting and approving or merging things that has been built by other developers. But at least open source is a way of encouraging people from very far locations to start contributing and be involved and have an opportunity to work for larger organizations that they may not have in the past. I think that's very exciting as well. So if we're going to wrap up now, could each one of you give sort of like a key takeaway from this conversation that you'd like the audience to think about or remember as they summarize? The topics we talked about covered a lot of grounds. Yeah, we covered a lot of things. That's the challenge. I would say that if I start, I would say your projects are getting more complex, so you cannot be an expert on everything. So relying on people that have significant expertise that will make the project successful is a key driver. Otherwise you will be lost in the middle of the forest looking for an exit. Yeah, I think too is like remembering that the things that you're doing are ultimately in service of your eventual customer. And that is easy to lose sight of if you're deep in the back end and your stuff doesn't necessarily get touched by their greasy little fingers. But there's plenty of things that we do there in service of making sure the customer experience at the end of the day is as good as it can be with the resources that we have and not introducing extra complexity if we don't have to, if it's not in service of the customer experience and making that better. My favorite part of DevOps is how much it increases security and how much it increases. So, like, a huge thing in security is availability and reliability, and DevOps just kind of supercharged that. And especially now we have platform engineering. And so many. I just feel like we're moving more and more towards making sure our services are always available and of higher quality with better integrity. And so as much as sometimes people are like, oh, we're doing this to serve the customer more, I'm just like, secretly in the back. I'm like, yeah. So I'm really happy about a lot of the things all of you hamed because I feel like we're all working towards the same goal and everyone in DevOps cares more about that than we ever did before with software development, from my perspective, I think, is really reminding ourselves that now software is in all aspects of human life and not being available and failure can have big consequences. So operations and keeping the software is running is really important. And almost like we, we should feel that burden in our shoulders, not forget it. Cool. Great. Okay, well, thank you everyone for coming today. This was great. It was so awesome meeting all of you. It's really good conversation. Yeah,