Conf42 Platform Engineering 2024 - Online

- premiere 5PM GMT

The Future of Platform Engineering

Video size:

Abstract

This talk explores the future of platform engineering, focusing on abstractions, standards, characteristics, and mechanisms. It discusses a decentralized platform approach and various platform levels.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello, everyone, and welcome to our session. Let me start this session with a bold statement. The next big revolution in platform engineering isn't about a new tool. It's about reimagining the entire concept of a platform. My name is Martin Zarkowski. I'm a senior solutions architect with AWS and have a long history in designing and also building platforms. And today with me is Marina Burkhardt, also solutions architect at AWS. And Marina has a strong DevOps background and experience in administrating and operating platforms. And we both, as solutions architects, are very much interested in building the platform together. of the future. So today we want to share our perspective on the future of platform engineering. But first, let's look back. Why are we doing this? Why are we building platforms? And what are platforms? Would I would define it as a group of technologies, that are used in as a base. And upon this base, upon this platform, we can develop application processes and other technologies. So we basically build some common components that handle a lot of generic stuff. And on top of that, we can build something that is special and unique. and we know this is a concept that has been used in many industries and they were very successful with this. using platforms to transformize their business. How does it work with software? let's think of the main goals of platforms in software development. It's about increasing the development speed and still meeting all the requirements that we get for our software. And in order to have this right balance between very high speed And high quality and meeting all the requirements, we need something that is helping us. We need abstractions. And often those abstractions come with standardizations. So platforms often enforce standards. And if I talk about standards, I often hear people saying that standards are taking away all the nice freedom and flexibility. Because people think of standards as limitations. But I would say, and this is a bold statement now, it's quite the opposite. Standards actually enable innovation. Standards help you to scale and standards remove all the friction. I want to give you a few examples where I believe that is true. So think of this highly standardized piece of paper. 210 by 297 millimeters. it's not roughly 200 by 300 millimeters. No. It is exactly 210 by 297 millimeters. by 297 millimeters, which is, by the way, exactly 1 16th of a square meter. And this standard 210 by 297 millimeters, it's called DIN A4. And I believe it's a great standard because our If we're talking about DNA4, no one has to argue how big that piece of paper will be. We can rely on a certain size. And we can build something that is, supporting that. we have fitting envelopes, we have folders, drawers, pockets. They all, we all know the right size for them. We can even build printers, fax machines, but we don't do that anymore. But, like all those, things that work together with that piece of paper. And yes, if we have that. Standardized piece of paper, we have to adjust to that standard. So if we write a letter, and I have this very long sentence in mind, I have to use a line break because of 210 by 297 millimeters. Or if I want to draw a picture, I have to accommodate that into this format. But think about it, does that really limit you in writing texts? Does that limit you in drawing pictures? But we have seen great texts, we have seen a lot of art. on a piece of paper, standardized piece of paper. And we have the same with technology. let me give you one example, very low level. we are using HTTP to make sure that our browsers Can talk to web servers, to other computers, and we are using this standard everywhere. So actually we can't imagine a world without that standard. yeah, there are some limitations. There are some standardizations in, in HTP. but this standard actually has enabled the internet and it never put a limit to the innovations that we are seeing today with the internet. So we have to realize a few things, locking things down and agreeing on a few things. can actually help us. It can boost innovation and it can boost creativity because those standards and abstractions, they help us to remove the cognitive overload we would have otherwise. So they help us with increasing the speed of development and they also help us to still meet all the requirements. So if we want to be successful with software development, We need kind of abstractions and standards, so we need those platforms. But there are many platforms and platform engineering is a term nowadays. So what makes a platform? A great platform. Maria will explain some of the key characteristics for a successful platform. there isn't a golden answer to Martin's question. But at least I brought you here three different characteristics that I think and consider that every single great platform should have. in order to be successful. So let's dive into those ones. As we just mentioned, platforms are going to come with abstractions. So that means abstractions and standards are going to be in place. And you might think that those are going to limit you, but actually totally the opposite. A platform should be an enabler. A platform should hide the complexity and should make it easier to use. And in order to achieve that, you should have a high level of transparency and make usage of shared components. So a platform shouldn't be a black box. A platform should have a high level of documentation so that every single person out there in the different teams can read that documentation, understand how the doc, understand how the components were built and which one was the intention of the usage of them. And also you can, make the code of the platform open source, at least internally for your different teams. And with a combination of two, of those two, those things, you're going to create a shared responsibility feeling across the different teams, which at the end is going to be beneficial for you. Nevertheless. you do not want to reinvent the wheel. So that means you would like to have those components that you built shared across the whole platform so that multiple teams can benefit of those components as well. Second characteristics, from great platform is that you would like that your platform is also extensible. So that means you do not have to force it. you should see which are the features that every single team is going to need in the future, but at least stay flexible in such a way that you can learn which one are the technologies that they are using, learn about the new tools that every team might implement or use from time to time, and also stay alert about different architectural patterns. There are trends in the industry and in the market out there. So that means you do not want to, force anyone to use a specific pattern, tool, technology, integration, but make and, or build the platform in such a way that it's attractive for multiple teams and all the teams actually to really make usage of your platform. When that happens. More and more teams are going to, use or want to implement things on top of your platform. And therefore you would like to be able to scale that platform as well. And that would be able just if you have a high level of automation in place. So you can use infrastructure as code, like for example, AWS, CDK. Thank you. to just make usage of the possibility of having a great automation and keep the pace of new teams onboarding in your platform. The third characteristic is that you would like that your platform is also evolutionary. So that means you do not want to have a platform that is static. You want something dynamic that keeps the pace of technology out there. With the time, you would like that this platform actually reduces the TCO, so the total cost of ownership, make it more efficient, and more, performant as well. And with the time, it should be easier and easier to use as well. How do we achieve this evolutionary characteristic? By treating the platform as a product. So that means having a product mindset. Nevertheless, something that experience taught me is that you can have the best intentions. You could have the best, specialist knowledgeable in a specific topic. But if you do not have mechanisms in place, there's no way that you can scale this across your organization and have the impact that you would like to have. to have. Therefore, I brought you four different characteristics of those mechanisms that I would invite you to think and consider when building and designing your platform. Being the first one about abstractions, so we want to build a platform in such a way that it abstracts. It has the complexity, but it has a high level of transparency so that everyone is going to be able to know how those components were built. You would like to have some same defaults, so having the possibility to set some, values per default. That helps the teams to reduce the mistakes. So reduce the errors, but at the same time have a high level of flexibility because you will going to, give them a set of configurations that are already properly set it up. The third abstraction is about automation. So you would like to keep the pace of technology, but increase the speed of every development team. and at the same time, reduce the friction of the processes. And last but not least, those platforms should actually help you with automation. Some control security things related. So we want to help you to, to enforce compliance measures and follow any regulation that you have to take in consideration, related to the market and the industry that you are part of while having this high level of share responsibility across the whole different teams. And the question here would be, whom is this platform actually going to enable? He said the platforms are very great way for centralizing expertise, but he also said you cannot centralize innovation. You have to leave that to the teams. that are the closest to the customer. Because those teams, they obviously have the best ideas and they will build innovation on top of a platform. Let's think about those teams that are doing the innovation. How do those innovative teams actually work? usually those teams are small and self sufficient. So that means they can do most or maybe all of their work independently from all the other teams that are also doing innovation. And they can make most or all of their decisions on their own. Let's think about this. Autonomies, teams on a platform, how does that work? Here we need to talk about the centralized versus decentralized approach. Let me start with the maybe more traditional centralized approach. We have a platform and that platform is designed and also operated by a platform team. and of course we learned platform needs to be evolutionary. so we have to add new features to our platform all the time. And in a perfect world, your platform team would anticipate every single feature request. And they would implement all the features. before someone even asks about the feature. But now let's look back to the reality. So in reality, it's different. In reality, app teams often request a new feature, and they want this feature as soon as possible because they need it for the innovation. And then your platform team needs to understand this new feature. And then the platform team needs to build the new feature. And all of this while operating a central platform that is growing a lot. So You're basically moving the cognitive load from the app team down to the platform team. And if you get more and more app teams doing this, what will happen? You will overload the platform team and your platform team will become your bottleneck. Because what are we doing here? We're not only centralizing expertise. We are also trying to centralize the platform innovation. How does it work for a decentralized platform approach? in that approach, the platform team is designing blueprints as a platform component. And all the app teams, they can use these blueprints and they can use them and create their own versions. of that. So we actually put a bit more responsibility into the hands of the developers. So the developers, the app teams, they own the entire lifecycle of the application, including all the platform components that they're using as a base. And while you could think maybe they have now the cognitive overload, but they don't get this cognitive overload because the platform components, the blueprints, they are using abstractions. that are provided by the platform team. So they basically don't have to reinvent everything and build it from scratch. They can build upon those abstractions. So they don't get this cognitive overload to understand the full stack completely from the beginning. So app teams can operate their own stack, but they still meet all the central requirements. And if required, they can evolve the stack, maybe on their own or together with the platform team. So what are we doing here? We are centralizing the expertise into those blueprints. But then we decentralize the innovation to the app teams. Let's think that a bit through. How would that work in reality? if we have this shared responsibility between app teams and platform teams. Who's doing what? Who is responsible for what part of this deck? Here we need to talk about ownership patterns and Marina will discuss a few of them. So there are different approaches and I brought you three of the ownership patterns that I've seen the most with our AWS customers in the way they have treated the productive workloads in their AWS accounts. By being the first one, Treating the whole AWS account itself as the platform. And the way you can imagine this is like by having an AWS account vending machine that is going to be owned by the platform engineering team, that it's going to provide AWS accounts. With already some, building blocks and components defined and deployed in the AWS account to every single team out there in your organization that needs an AWS account in order to run the resources needed for your business. The different projects and use cases, and those components might be related to networking, observability, security, or anything out there that comes to your mind, where the platform engineering team is going to, for the networking, for example, design the whole network, like a network. how many VPCs is going to, that account is going to have, how many subnets? What's, how does the routing table, for example, look like, and so on. Related to observability, you might have already some, I don't know, logs defines, metrics, some dashboard, all of the, all of this may be in CloudWatch. And also specify and already implement some security mechanisms in that account as well. How can this platform engineering build such a vending machine through, for example, the usage of AWS, organizations, AWS config, CloudWatch and IIM, for example, as well. The second ownership pattern is related to managed clusters, where in the other approach, you had like a higher level of flexibility, But lower level of abstraction that might not really be needed by every single team. Like for example, a team that runs some containers, would not really care about what's the VPC that it's already there in place in the AWS account, but instead having a possibility. to just run their containers in a cluster that it's already managed by another team. So that means they still would like to have those standards components to networking and observability, but tailored specifically for a managed cluster. So for example, They would like to have some health checks, or maybe have the possibility to monitor how many containers are running the, in the whole cluster itself, or maybe have some possibility to, manage the users that are going to have access to the cluster and also some input and output rules for the whole. container itself that it's running there. So this option is going to have a lower level of flexibility, but a higher level of abstraction. And the third ownership pattern is related to deployable application patterns. So this is like providing blueprints, like providing some AWS CDK constructs. this might. for you not resonate as a platform and rather like blueprints and guidelines on how to deploy things but this is yet another ownership pattern for a platform itself and this is pretty much a modern way also of treating it than I also seen A lot of times we're actually the teams that might benefit the most of T of this are the high performing teams are the ones that are already used to work with, for example, AWS accounts and need some blueprints that helps them to deploy some applications faster with some standards that are well defined there with some architectural patterns as well. That might help them to reduce the development time and improve already, the, the possibility of building something on top as well. Having now those three ownership patterns in mind, I would like to hand over to Martin, who is going to actually explain how by having the possibility and the freedom of different patterns, you do not have to force. How can I use any team in order to implement and make usage of this platform? you don't have to decide exclusively what pattern to use. You get the freedom of choice. Let me show you how that could work. Let's imagine we have this platform team and they're dealing with, let's say, four teams. These are decentralized dev teams. first of all of our teams get those pre configured accounts from the account vending machine. And, team one is perfectly fine with that account. So they get this pre configured account as a base level platform, and they build everything else on top of that, completely on their own. team two is different. They use those managed clusters. And on top of that, they also want to have those deployable patterns, maybe in a cluster that could be a hand chart. team three, deploys the application into a managed cluster, but they don't need all those patterns on top. They have their own way of doing things. And team four, let's say they're the serverless game. So they use those templated patterns for serverless applications in their pre configured account. now we have four different ways of doing things. Let's reimagine the term platform. All of this is your platform. Some parts of your platform are more central, like the account vending or maybe the managed cluster. And some parts are more decentralized, like all those deployable application patterns and so on. And by giving all the teams the freedom of choice, you don't have to force a team to follow one approach. That might work for you, but that wouldn't work for me. for them. So the idea is to make a platform not too restrictive, but make the platform open to innovation and to development. And I can recommend working towards those decentralized teams with shared patterns and services as a platform. And you can see those blueprints that are shared on your platform as a golden path. Maybe this concept of a golden path is very well known. Maybe also the concept of a paved road. And let's see, those blueprints as a golden path. On a paved road. While this is the golden path, there's still a bit of room to do things differently. On a paved road. So that's your opinionated and supported way. of using technology. this is the way to make it for the teams as easy as possible to use technology and to use your platform. But we want to be flexible. if someone has an alternative idea, let them go but left or right from that golden path on that still paved road still, to a certain degree. there are still guardrails, outside of that paved road. Maybe. this idea will open another road, maybe a new shortcut, maybe, this improvement will even be your new golden path. So basically treat your platform as a product, make it similar to your product development, have a product mindset when building your platform. how can you do that? we can learn from the development of open source. products because they follow the same principles. So whenever you build something for your platform, make it versioned. So a version piece of software that can be evolved by the user community, maybe make the platform team as the main contributor, but allow other ones to contribute and create their own version, make it decentralized. So every component should have an independent life cycle that makes the scaling much easier because you have smaller pieces. And, it also reduced the blast radius and dependencies and so on and make it usable. So if you have this product mindset that we just discussed, you will create good accessibility like good documentation, scaffolding, a CLI, a UI integration, and so on. And then make it customizable. So be open to changes. Allow someone to parameterize your product. your platform components, allow to choose escape hatches if possible. Let them fork it, let them change it if they need. Don't build a black box, make it transparent. Make it a white box. Don't hide the implementation. of your platform because people rely on it and then make it self service. so everything should be fully automated as much as possible. So everything can be requested and deployed via APIs. So that is an evolutionary platform guided by the principles of open source development. And one last thing to mention here. Always stay ahead with your platform. So we discussed this, a platform will always build on a certain base, like the AWS cloud, for instance. And we also discussed that your platform should evolve. But here's one thing to remember, the base that you built your platform on does also evolve. So if you don't evolve your platform, the things that you're building on evolve on top of that base. And you don't adapt it to the base. you will have a syncing platform, a platform that has no true benefits anymore. Maybe the platform even makes it more difficult to do things, compared to the native approach. So that will be a platform was a burden and no one wants to use that. So if you always evolve your platform and you always adapt it to the base. You will have something that we would call a floating platform, a platform that is always ahead of the base. And that is a platform that offers true benefits to all the users of the platform. So a platform that enables all your teams. So our recommendation is always build floating platforms. What did we learn today? So standards are not limitations. Often I would say they are the key to innovation. So don't build a platform for the sake of it. of having a platform for the sake of doing platform engineering. Build a platform that enables your teams. A platform that is extensible and evolutionary. And always think about decentralization of the platform with centralized blueprints. And if you do that, make it as flexible as possible. Build a golden path, but be flexible. And always Stay ahead with your platform. Thanks a lot for listening to our talk today. we really value your feedback. there are two links, how you can reach us. we're happy to discuss that concept even further with you. And if you have some feedback for us, if you liked what we told you today, there's a feedback link. We're happy to hear your feedback. It only takes 15 seconds to fill out the survey. Thank you so much for joining us today.
...

Marina Burkhardt

Global Solutions Architect @ AWS

Marina Burkhardt's LinkedIn account Marina Burkhardt's twitter account

Martin Sakowski

Senior Solutions Architect @ AWS

Martin Sakowski's LinkedIn account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)