Conf42 Platform Engineering 2024 - Online

- premiere 5PM GMT

Platform Engineering Untold Truths: Is It Just an Infrastructure Matter?

Video size:

Abstract

To streamline delivery, companies need a structured approach through platform engineering. This approach goes beyond just infrastructure. Effective platforms must focus on data, composability, and the entire organizational framework. Let’s explore the core principles for successful platform engineering.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello, everyone, and welcome to my session about platform engineering untold truths. For sure, many of you are already familiar with the classic narrative behind the concept of platform engineering, where infrastructure orchestration is at the core of every platform. In this talk, I want to present a new way of looking at platform. It's not that the current view is wrong, but rather we'll try to extend what we commonly refer to as platforms today to include other important assets for today's digital company. Let me introduce myself briefly. My name is Graziano from Italy. I'm a Developer Relations Engineer at MIA Platform, and I focus on sharing the wonderful things the cloud native world has to offer. At MIA Platform, we offer a digital platform builder to the market, which is one of the tools available for building what we will be discussing today. If you'd like to stay in touch with me, you can find me on LinkedIn or GitHub, and if you are interested to, In this and other cloud native topics, I invite you to visit the NeoPlatform blog or my website at GrazianoCast. com. Don't worry, I'll provide links to all my contacts in the final feedback form. Let's start with a somewhat provocative statement. Platform engineering is the natural evolution of DevOps. Don't get me wrong, I'm not saying that to DevOps long life platform engineering. In fact, DevOps practices are more alive than ever. And it's thanks to this vitality that platform engineering is gaining momentum today. What I mean by this statement is that platform engineering emerges as a methodology to industrialize the practices, making them more accessible and practical. But why? Why do we need to make it more accessible? What's the point of making something accessible that worked well until recently? After all, platform engineering has only been discussed for a couple of years, while DevOps has been around since 2007. The typical phrase you hear when talking about DevOps is You build it, you run it. Coined by Werner Vogels, CTO of Amazon, in 2006. He explained how putting developers in front of operational challenges brought numerous benefits to Amazon's production chain. However, a lot of A lot has changed since 2006, including technology, but then we dealt with virtual machine and were far from tools like Kubernetes and other subsequent wave of cloud native technologies to give you an idea today. The CNCF landscape includes 119 projects and more than 1, 800 code repositories, ranging from libraries and frameworks to utilities and more. The increasing technological capabilities at our disposal makes it seem like a huge advantage. At first glance, however, being surrounded by a multitude of tools that often do similar or completely different things get actually become an obstacle to innovation. Imagine you are on a hike following a straight path for hours. Perhaps interrupted by a few folks in the road. Suddenly you find yourself facing hundreds of different paths all at once. You might think what an adventure, but you'd probably also admit that getting lost in such a scenario is a real risk. This abundance of options can hinder innovation. To illustrate this concept, I used a simple example, but there are psychological theories on cognitive load that highlight these aspects. Cognitive load is not something to underestimate, especially for so called knowledge workers, which includes developers and anyone with a job title related to the technological field. This explains why platform engineering emerged and why the adoption of platform of cloud native technologies and the box practices, of course, needs to be industrialized in this. In this slide, you can see a quote from Gartner paper that summarizes this concept. the platform engineering to shift the burden from the platform users, of course, Who are the developers by uploading it onto a tool that is the platform itself. The EDP or internal developer platform is the tangible result of applying platform engineering. To understand what IDP is and what benefit it brings, let first explore what a platform is in its most general sense. Think of an organization whose business is to create software. Typically, these organizations are structured into development teams that vary in size and, are often cross functional to some extent. Typically, the organization grows, or it dies. If it's business includes involves multiple teams, each team will have its own ownership off their respective product. Now, it's easy to imagine that in such a company, even though different teams to work on different products, they still share a series of For example, all teams will have boundaries set by the higher ranks of the organization. Therefore, it's likely that all applications will use the same type of database, be built with a pool of two or three programming languages or frameworks, and use common monitoring tools and libraries, for example. All these common aspects within an organization, when taken. Rationalized and combined using an as a service approach in a common and accessible place for everyone within the organization constitute the platform. The goal of applying these methodologies to provide the developers with everything they need to do their jobs in a self service manner without having to deal with the countless service requests or inventing the wheel. This image illustrates the concept. translate, what we have discussed to the experiences of going the grocery shopping. In this example, we are using platform engineering when, we know what we need and we can take it from the shelves, go to the self checkout, pay and live completely on your own. We are not using it if, When we get to the checkout, despite knowing what we need, we have to wait in line and rely on someone else to enable us to pay. Now, everything we have talked about doesn't grow on trees and certainly isn't something that can be done in a day. We can also rationalize the role involved within the platform and divide them into Three main groups. The first groups are the creators, and these are those who initially adopt a technology or build something that will be later made available for everyone to reuse. They create building blocks that are then used to construct more complex products. They can also be considered and re adopted. There's especially if building these blocks involves experimenting with new tools and technologies for the organization. The second groups are the composer, which are those who benefits from the building blocks provided within the platform to create digital products or even more complex systems. They, there isn't a strict separation between the creators and composer. The work of a composer today can become the building block of for someone else tomorrow, making them a creator as well. Finally, there are the curators of platform engineering. They simply, sorry, they simplify the cooperation between creators and composer by providing tools, making the work of creator accessible, sharing best practices and more. The cooperation between these three roles ensure that the platform is a place where the values is created to understand this. Let's take the example of a furniture making. Okay. The platform is our woodworking shop and the curator or platform engineers are the ones who ensure that all the tools in the shop are properly Perfectly functioning for what the user will need to do. they bring in new tools and create best practices for the use. For example, there are, then there are the carpenters who use these tools to create pieces of four meters. For example, here you can see a drawer. They are the creators. our platform stands to this example. These drawers is then taken by other carpenters who in this case act as composer and they create a chest of drawers, for example. And at this point, the work of this last group of carpenters can become a Building block itself, since the newly assembled chest of drawers can be used to build something more complex, like a kitchen, just create a vicious cycle of value and sharing within the platform. Translating this example in the world of software development, the individual building block provided by platform engineering could be, for example, I don't know, a Kubernetes cluster with all necessary configuration, a cross plane CRD, a Kafka topic, an API to perform payments, or even a completely new application accessible to users, for example. That woodworking shop is our EDP, and the facilities it offers could be a catalog where all composable resources are available. A monitoring system to track all the activities of all the microservices are CLI to view log introduction, for example, and many others. This cycle can be seen as a circular economy in software development, where resources are continuously reused and repurposed, leading to a more efficient and sustainable development process. We know what a platform is, why we need it and how to use it to create value. But in the end, what does this platform look like? First, let me say that, it is, Much more complex than what I'm about to present, because each adoption of this methodology is tailored to the adoption to the adopters need, making each journey unique with specific characteristics that vary from use case to use case. However, let's try to provide a good abstraction of what. can be considered a platform. First of all, we have our software delivery pipeline divided into product teams that build a digital product on one side. On the other hand, there are various dependencies that these product teams need to manage, including, for example, infrastructure, cloud providers, and other such systems, and so on. Our platform sits between the delivery teams and their dependencies, and is made up of various components. First of all, there are several interfaces that allow us to interact with the platform on multiple levels. Imagine, for example, a Web UI, a set of APIs, a CLI, and so on. Then there is the heart of the platform, the control plane, which is responsible for orchestrating all the processes within the platform. And finally, we have the components that interact with the infrastructure provider. Allowing developers to have runtime environments on demand, for example, by clicking by simply clicking on a button on a web UI without having to open service request and wait for someone to intervene. Or there is the part. of the platform that handles orchestration and simplified adoption of DevOps tools, such as, for example, pipeline. This includes providing pre built pipeline to integrate into your system, tracing and monitoring systems, secret management, and testing. For example, it could provide access to direct time with Intel. telepresence, for example. And so in this context, we will have, cloud engineers who will create automation for the infrastructure providers. This ensure that, this services can be used in a self service manner with as little friction as possible for the developers. Developers use the platform and the capability it provides to build digital products. And finally, we are, and finally there are the platform engineers who orchestrate the work of the previous two roles and ensure that everything run as smoothly as possible. For example, they define best practices, golden path, documentation of advocates for the platform and handle all aspects that fall under the umbrella of platform as a product methodology. And this is the most common representation of what a platform is. For example, I recommend looking at the platform web paper by CNCF, which presents platform in a very similar way. Platform engineering is often only associated with infrastructure and DevOps simplification, with the primary goal of providing developers with a self service platform. But is that enough to cover the entire value chain to efficiently deliver a product and application? The point is that by focusing too much on applications, And the tools around them, we overlook the fact that today's software is built on much more than just run time it's run on. While the current platform engineering practices efficaciously address many aspects of simplifying people's lives, there are other areas that require attention. Data, ML, API. Security, privacy and others are crucial for better software product lifecycle by relying platform engineering only to the infrastructure aspects. We run the risk of creating the new barriers between platform engineers and developers similar to the barriers that exist in the past between operations and developers. Digital products as they are crafted today are based on data, business logic, and the know how and experience gained by the organization, both from technological and business domain perspective. These are important assets that must be included and valued within the platform as composable resources that can be leveraged to generate value. With the representation of platform, we had focused on the automation aspect of the infrastructure, which was central to the sheen. What we aim to do with this different vision of platform is to give space to and value these assets that was mentioned before. The first added benefit is composability. Both in terms of application accessibility and application composability. By accessibility, we can range from an API gateway to access the application endpoint, to portals that expose documentation and describe capabilities, making our software usable from the outside, for example. This also includes composable logic for solving a business problem, which can involve using, for example, orchestrators as building blocks to manage sagas in a standardized way, creating and managing the microphone tents or providing a marketplace within the platform for elements that can be used to build a more complex solution. Last but not least are the data. A digital product is unlikely to succeed without a data platform. Therefore, the platform should include tools like, data catalog to improve data discoverability, a control plane to monitor data flows in real time from source to destination, allowing for stopping, resuming and analyzing these flows. And providing an approach for building data fabric within the platform by adding these two elements. We achieve a more modern vision of a platform that includes all the resources related to creating digital products. Now, the million dollar question. We are in the boom of artificial intelligence. Could I avoid bringing up AI? Of course not. I'll just spend a few words to introduce what AI is today and what its future prospects are in this context. The vision I want to present, I want to present is not much about how platforms are useful Or will be for building AI powered application, but rather a perspective where AI is one of the interfaces that platform provides us with an intelligent yet simple interface that allow us to use natural language, we can elevate the platform user experience and consequently the developer experience to a whole new level from simple device to conversational device. Imagine having a virtual assistant that speaks like you and respond as you would with the added benefit of perfectly understanding the context of your platform because it is your platform. Now, imagine what you could do with such an assistant that know your data models. They are lineage and the pipeline that managed that, it knows the tools you can use the best practices for using them and they call them path designed to help you assume designed to help you use. them optimally within the platform context. It also know every item in the marketplace of building blocks you can use and can provide advice on how to integrate them, including how others have done so. You can see that, I don't know, troubleshooting production issues or conducting a POC in such a context is on a whole different level than now. We have reached the end of the presentation and I'd like to recap what we have talked about. Covered and what I hope you have taken away from this session first, the software industry and the digital product creation sector are undergoing a transformation. We have moved from project with outcomes that could be afford to be isolated to the necessity of having everything be integrable and integrated within a clear and defined context. This is where the need of platform arises. Second, and this is the core of this talk, platform engineering encompasses more aspects of digital application creation, not just infrastructure and DevOps. Finally, I don't know if artificial intelligence, we take our jobs, but, I, but I think that if integrated into our platform, it can elevate the developer experience. So we provide to our decks to an entirely new level. So let's give credit where credit is due. First of all, I want to mention my CTO, Giulio Ruggiero, who was the first to formulate these concepts and present them during the keynote at Platformosphere, the Italian conference dedicated to end development of digital platforms. Then, this concept were also brought to the platform working group of the CNCF and just, Some weeks ago, an article was published and that delves deeper into the topics we have discussed today, and so I highly recommend you to read it. Thank you for being here with me to discuss these topics. Here you will also find my contact information if you want to stay in touch with me. And there is a dedicated space for questions on the topic, which I'll try to answer on LinkedIn or via email privately. Additionally, you can find my website and the platform blog where there are articles and other talks on this and other more or less technical topics in the cloud native world. Last but not least, in the feedback form, you can find direct link to all resources mentioned during this presentation. Thank you all for listening to me and a special thanks to Conf42 organizer for having me today. Bye.
...

Graziano Casto

Developer Relations Engineer @ Mia-Platform

Graziano Casto's LinkedIn account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)