In his bestselling 2014 book, Homo Sapiens Yuval Noir, Harare discusses a brief history of humankind and one of the most important points in this book is that of stories. Stories and words are the master key that unlock human potential, whether that be your religion, your nationality, or even the company you work for and the values that that company has. Our ability to communicate an idea, to get people behind it, is something that really differentiates us from the animals. Now, if you've just joined this late or you're kind of confused right now because you thought this was a talk on event driven architecture, that's okay, because this is related to event driven architecture. And that's because Eda is fundamentally a communication pattern, a way of modelling the communication between your systems, whether that be the technical communication between two services or the human communication. As you talk to each other about what it is that your system does, the language, the story of your business, drives its technical implementation. And that's what you're going to learn about in this talk. You're going to learn the secrets of event driven architecture. How you can build event driven systems from the basic concepts through to some more intermediate patterns. I'm James Eastham. I'm a senior cloud architect at AWS and I want to set the expectation really early on. Now, this is not a talk about AWS, about services. This is a simple talk that's simply about boxes and lines. Yeah, it's one of those kind of talks. And I want to start by being really honest with you all. I've built event driven systems in the past and I've got them horribly, horribly wrong. Which might make you all think, why would you want to learn from me, someone who's got it wrong so many times? One of my old mentors always said to me, it's failure that builds experience. You learn so much more in failure than you do in success. So that's why I'm here to walk you through today some practical tips and advice from things I've learned from building event driven systems in the real world. Now, if any of you have seen me talk before, you'll know that there's one thing I love talking about almost as much as serverless and event driven architecture. And that is, of course, pizza. And for the rest of this talk, you all have a task. You all need to put on your imagination hats, because you're now developers working for a completely fictional pizza delivery company called Plant based Pizza. And you're going to introduce an event driven architecture to the plant based pizza system you've already adopted microservices, and all your interservice communication right now happens using HTTP APIs. And this is your architecture today. As you can see, all of the services need to know where all of the other services are. The order processing service is at the core of your system. This is your core domain. If people can't order pizzas, then you don't really have a business as a pizza delivery company. It has really explicit dependencies to the kitchen, the pickup and the delivery service. And now you also need to add a new service, a loyalty point service. And the order service needs to know where all of these services are at runtime. And this is of course a form of coupling. And I'm sure you've all heard the general rule of thumb in software design to aim for low coupling and high cohesion. And cohesion is a relatively straightforward one. Things that change together should live together, and you've already got that in your microservices architecture. You've broken your system down into these small, independently deployable units that are all focused on one job and doing that one job really, really well. Coupling is a little bit more nuanced, however, and that's because there's many different types of coupling. You've got technology coupling. Are you using Java net C plus plus rust? You've got temporal location coupling, runtime coupling. What's the IP address of the service I need to talk to? What's its DNS name? You've got data format, semantic and data type coupling. Are you using binary JSOn protobuff, Ints strings? How do you handle null values? What casing are you using on your json payloads? And some of these are a lot more tricky than others. Primarily HTTP communication is dealing with runtime and location coupling. At runtime, your services need to know the location of all the other services that they need to communicate with. And coupling isn't necessarily a bad thing, it's just something that you need to deal with. Without coupling, it would be really hard to get things done. And I really like this quote from Gregor Hopi. The appropriate level of coupling depends on the level of control you have over the endpoints. And what I think Gregor means there is that if you're a team building three or four different microservices, coupling them services together more tightly isn't necessarily a bad idea because you control the release cycles, the SLAs, the schemas, the deployments. But once you start integrating with something outside of your control, that might be a third party API, that might be another team inside your organization, you want to reduce that coupling as much AWS possible. So if it's coupling that we're working with here, why does it help to be event driven? What does that actually give you? What benefits does that have? Because let's have a look at the challenges you face with your current microservices architecture. As of right now, you need to create your new loyalty point service. But then to do that, you also need to make changes to your order processing service. And it's becoming really frustrating because every time you need to make a change, you need to then go back and also make a change to the order processing service. And actually, when you think about this, should the order processing service even care about the fact that people are collecting loyalty points? Is it the job of the order processing service? The answer to that is probably not because of this coupling. A failure in your loyalty point service, something that isn't on the critical path of your system functionality can also cause the knock on effect of taking other services offline. So now if your loyalty service fails, you could also then lose the ability to collect orders. Probably not something you want to do now. Of course, I realize I'm oversimplifying this massively. There are ways you can build microservices that handle all of these problems. Circuit breakers, backoffs, proper error handling. Just because you are building microservices in this way doesn't mean that your system is immediately going to be unstable. So if that is why you might want to be event driven, what does it actually mean to be event driven? The most succinct way I've seen of putting this is from one of Martin Fowler's talks on this topic about event driven architecture. At its core, event driven architecture is about reversing dependencies, about reversing all of these lines of integration. Now, your order service doesn't need to know anything about any other service consuming from it. And by the end of this talk, you're going to have made this entire flow event driven. But first, let's get a little bit theoretical for a moment. And I promise this theoretical bit won't take long and just level set on some of the key ideas in event driven architecture, the most fundamental, of course, what is an event? In my mind, an event is an immutable fact. It's something that's happened in the past. It cannot be changed. It has happened. The example I like to use here is that of a light switch. If you walk into a room in the middle of the night and you want to avoid all the Lego blicks that your kids have left all over the floor so you don't hurt your feet. So you want to turn on the light, you want to be able to see where you're going. So you hit the lights switch and that's going to raise a light switched on event. And at that moment in time it is impossible to unswitch on the light. You can't unswitched it on. The only way to remove the light from the room and run the gauntlet of Lego bricks on the floor is to hit the light switch again, which will then raise a light switched off. Light switched on, light switched off. Then things happen. They can't be changed. It's immutable. So if that's the definition of an event, what does it mean to be event driven? And I actually think there's some confusion here sometimes between event driven systems and what I'm starting to call event based systems. And there's an important distinction here. Both types of systems trigger functionality using events, but in an event driven system, events are business events. An event driven system triggers your business functionality from events that are related to your business. Triggering functionality in software with events is not a new thing. I'm a. Net developer. I started building with windows forms and in windows forms and net you can actually trigger functionality on button clicked events, mouse hovered, window resized. All of these things are events. That doesn't mean you're building an event driven system. And you can even apply this to cloud native ways of building things. If you're using Amazon s three, for example, that when you drop a file into an S three bucket, that will raise an S three object created event. But that's a technical event, not a business event. Business events are things like pizza boxed or replaced. Staff member clocked in. You could look at two businesses in the exact same domain, two different pizza restaurants, both building event driven systems. And they will have a completely different set of events because they have different terminology, a different language. Now, they might have similarities in the things that are event based. Button clicked s three object created. And this comes back to the idea of stories of the words you use in your business. An event driven architecture lets you tell the story of your business through its technical implementation. So another thing you'll commonly hear when you start learning about event driven architecture is that events are first class citizens. And that might make you huh, what does that mean? Well, as well as these events being based on your business events, you should also treat your events like an API. I'm sure many of you listening are familiar with the idea of API first design, the idea of API first design, if you're not, is that the first thing you do when building a system is to design its API. And then when it comes to integrating things together, you have an agreed contract that everyone's aware of. So making events first class citizens means focusing on event first design. Your event schema is your API in event driven systems. It's the language your services use to talk to each other. So when you're thinking about events and the schema of them events, it's not as simple as just starting to fling JSon round all over the place. There are different types of events that you might see, and it's important to really think this through carefully, because how you structure your events can be a deal breaker for the success of your system over time. Remember, the events are your API. The first type of event you'll commonly encounter are notification events or sparse events. These are really simple, small packages of data that simply notify another system that something has happened. And this is where you start with your first implementation of event driven architecture at plant based pizza. Let's zoom in on that implementation. The integration between the order processing service and the kitchen service. So now your order service is just publishing an order confirmed event and the kitchen service is listening for that event. And it says, thank you very much. I know that this order has been confirmed, order 1234. And then it thinks, I don't actually know what I need to create. I don't actually know what pizzas people have ordered. All I have is this order confirmed event with an order ID. So what I'm going to do is I'm going to call back, I'm going to reach back to the order processing service, to the API. It's exposed and I'm going to grab the order details. And at this point you might celebrate, because now you're event driven, right? But ask yourself, is this actually any better? You're still coupled, you're still dependent. And now actually your order service needs to handle all this, Lord, from all these different places. This is a pattern I've built before and I've done exactly this. And it was wonderful to begin with. And then there was another subscriber to the order service, to the order confirmed event, and then three more, and then another one, and then a few more, and then maybe just some more. And then eventually what happens is that your order processing service sets on fire because it can't deal with all this, Lord, coming from all these other places. A core principle of event driven architecture is that services are unaware of the downstream consumers of their events. So whilst this pattern works from a functional perspective, you open your producer up to receiving potentially infinite numbers of events from other systems. So is there a better way? Is there another option? And this leads us nicely onto the second type of event, and that is event carriage state transfer, or also what is known as a more fat event. And let's actually take an example here. This is the net representation of the notification event. And for any of you non net developers out there, I've also got a json version. So imagine this as your notification event. It's an order confirmed event that simply includes the order ID. And this is what that same event might look like as event carried state transfer or as a more fat event. And what you see is that the state of the order is carried as part of the event payload. This means that any consuming service has all the data it needs to actually create the order. The kitchen service in this case knows that it needs to create lots of margarita pizzas. Now that you understand these different types of events, you might be wondering which is the right one to pick. Hold that thought, we're going to stay theoretical just for a few minutes longer. This is the simplest structure of an event driven system, primarily made up of three different parts. An event producer, a central broker of some kind, and then some kind of event consumer. And these events are typically passed around using an event broker, but also queues and topics and streams. And what you're immediately going to notice with this is that the producer and consumer have no idea that each other exists. They aren't communicating with each other directly, which almost immediately reduces your runtime coupling and completely removes your location coupling. Communication is asynchronous. First, and the primary integration pattern at play here is the publish subscribe pattern. Of course, not all coupling is removed. As you've just learned, your event schema is the contract you agree on. So you've got semantic coupling at the level of the event schema. And this is the largest amount of coupling you're typically going to see in an event driven architecture. Which is why you need to be so careful with how you design your schemas. As a producer or a publisher, you have some responsibilities, and the biggest responsibility you have is to agree and stay consistent to that agreed event schema. Don't just be changing things willy nilly and you want to ensure that you keep backwards compatibility wherever possible. What you're not responsible for is how your events are used. You don't care, you're just publishing this event out to the world. And the other thing you're not responsible for, which is particularly interesting when compared with HTTP communication, is that you don't now need to care about the constraints of the system you're integrating with. If you think about building things with HTTP APIs, you need to care about the authentication for that API, you need to care about the rate limits, its throughput, how much load can it deal with over time, what the requests per second. Whereas now when you're building an event driven system, you don't need to worry about that. That responsibility shifts to the responsibility of the subscriber. As a subscriber, you're now in charge of controlling the load on your own system, and it's a much more logical way of things being controlled. You control the Lord on your own system. One of the challenges with this though, is that you can still overload your system as a consumer. If one of the upstream producers suddenly starts publishing 100 times, 200 times more events, that's still going to overload your system. And a pattern that can help here is using some kind of queue, some kind of durable storage at the boundary of your system. So instead of subscribing your subscriber directly to the event bus, you might instead subscribe a queue to the event bus. And that means when that 100 times, 200 times Lord comes through the system, it's all going to build up in the queue. And that protects your system. It acts as a buffer against your system and you can then work off that queue as and when you need to. This functions much like an anticorruption layer. It allows you to validate the schema of the events coming in and generally just be more defensive with your programming without risking losing or dropping data. There's another really interesting and slightly more subtle benefit with event driven architecture, and that's based on the fact that EDAs are built using business events. Now I'm sure any of you who've worked in software for any amount of time have had problems with communication. You've been talking to a business expert in your organization, and as soon as you mention the word HTTP, microservice or service, they instantly shut down. But when system functionality is driven by events, you're now using a language that's familiar to business people. I could go and have a conversation with a business expert and ask them what should happen after an order gets canceled, and then you can map that exact language into your actual system. In much the same way, domain driven design gave us a ubiquitous language. We're now taking this one step further and having that language drive business functionality. Because remember, event driven architecture is fundamentally a communication pattern, a way of modeling the integration between your systems. Now, one of the commonly touted benefits of event driven architecture is evolvability. Having this central stream of events that any consumer can hook into and that allows you to add and remove services without affecting other parts of your architecture is wonderful. It sounds great, sounds like a nirvana of software design, but you can get yourself in some trouble here. And this comes back to this problem with schema design. Here's a very similar problem to something that I've encountered in my career. Here's an event that I was publishing and what happens now? If you need to change the schema of this event, and this is what you need to do in plant based pizza, you've gone multi currency, it's blowing up, people love your pizzas and you've gone global. So you now need to start including the currency as part of the order value in your order confirmed event. So what you do is you just update the event. You change the value property to be a value. And the currency, it's now an object, because it's only the kitchen service that cares about this event. Nobody else really cares about that. And then you realize that somebody had added an analytics service, and that analytics service is very, very interested in that order value property. And now by making this breaking change, you've just broken a system that you didn't even know existed, which doesn't sound particularly evolvable to me. And this is one of the challenges you'll face. With these fatter, more event carried state transfer type events. You reduce the need for a consumer to call back, fantastic. But now you're coupled at a schema level, and this will definitely lead you to be asking the question, what the heck do I do now? And I'm going to put a word on the screen now, and this word is going to give you all the shivers. You're all going to get a bit tingly, you're going to feel awful. It's a really naughty word. Please stick with me. I promise it won't be on the screen for long. That word is governance. Yeah, boring, horrible, right? It gives you all feelings of red tape and bureaucracy. But governance is really important in event driven systems. And this doesn't need to be painful. There doesn't need to be bureaucracy and red tape. You can simply use a process that I'm sure many of you are familiar with, and that's the process of the RFC, or the request for comment whenever you want. To make a breaking change to your events, you simply introduce an RFC and then you get the input of other key stakeholders in your business. And you can do this completely asynchronously. All your teams can stay decoupled, but you're still keeping an element of collaboration and knowledge sharing. Once you've gone through the RFC process, then you can actually manage this at a technical level as well. You can introduce a pattern like the metadata data pattern. You break your event payload into two separate parts. You have a really clearly defined schema. An alternate approach to metadata data is to use the cloud events specification. The cloud event specification is a really formal defined specification for communicating between different systems and event driven architectures. And what makes this really powerful is some of these metadata fields that get included. You'll notice I've got a type and a source, an event ID, a uniquely generated event identifier that's generated at the point of publish to then allow downstream consumers to implement things like item potency to understand if it's seen an event before. You've also got the time, you've got the actual data, the actual event payload, and one of the most important properties in here is the version having a version number in your events. The ability to version things is vital as you start building event driven systems. And let's have a look what this could look like in practice. Right now your order processing service is publishing version one of your order confirmed event. And when the kitchen service defined its subscription, it subscribed to version one, specifically of the order confirmed event. And then when you need to make the breaking change, you'll start publishing version two of the order confirmed event. And notice you are still also publishing version one. So at this moment in time you're publishing version two. And version two is incredibly sad and upset because there's no one actually listening to it. You can then put out a message to the rest of your organization saying hey everybody, can you switch over to version two? And over time all of your downstream consumers are going to switch over and start working with version two of the event. Eventually you can then turn off version one. And that brings us back to the question, fat versus spice events. Which one is the one to use? And of course this is a software conference, so the answer to this is always, it depends. Personally, I've gone backwards and forwards with this argument so many times in my own head. Fatter events, combined with some kind of metadata pattern that allows versioning, really eases the evolvability. It reduces coupling and it also removes the need for a callback. What it does give you, however, is really tight coupling at the schema level. You need to be really careful whenever you introduce breaking changes, and without the right governance and versioning in place, that can cause you problems over time, which then might make you think, well, long live the sparse event. Only ever publish sparse events. And honestly, even as I think about this now, my opinion on this starts to change. To take this sparse event idea to its absolute extreme, you could publish events that only ever include identifiers, and you maybe only add additional properties as and when additional people ask for them. Remember, one of the core principles of microservice is that each service owns its own data. So just publishing identifiers allows each service to stay in control of its own data, and this gives you the purest amount of looser coupling. If you never allow services to call back and gather more data, it forces the rest of your system to act in a really specific way. However, this is a pretty impractical approach. You're almost always going to run into some kind of issues if you try and stick to this rigid sparse event where you only ever include identifier. However, an additional option I've been exploring is actually to still only include identifiers, but include a wider range of identifiers. So as well as publishing the order ID, which is something that the order processing service owns, you also publish the customer ID and the delivery address ID. And if you think about this from a logical perspective, as I'm on my phone trying to order my pizza, the first thing I'm going to do is I'm going to add my delivery address. And then when I actually come to submit my order, I'm going to select the delivery address that I want to use. That means that the order service is going to receive the ID for my delivery address. I can publish that as part of the event payload, and then my delivery service now knows which delivery address to deliver this order to. The really key takeaway here, talking about evolvability, is to embrace that metadata data pattern. Be incredibly intentional about how you structure your events. Now, as you've seen over these last few diagrams, one of the big things you'll need to embrace with Eventdriven architecture is that your systems are going to be asynchronous. You need to think about every part of your system interacting in an asynchronous way. And if you're a developer who's used to making method calls or HTTP calls, and then waiting for that call to return, this can be a bit of a paradigm shift. Although I really struggled with this when I started building event driven systems. Now your service is doing a piece of work, maybe storing some data, and then it's just going to publish an event. And then maybe sometime, potentially, possibly later, your service is going to receive that event back. Let's think about this same flow again. The order processing service publishes an order confirmed event and some point somewhere, hopefully, eventually, it's going to receive an order cooked event when the kitchen has finished doing its work. But you might not. There's no way to know if that event is ever going to come back. Frankly, there's no way to know if the kitchen ever even receives that order confirmed event. And this can lead to a challenge around consistency. Asynchronous systems naturally lead you to need to embrace eventual consistency. And there's two types of consistency. There's strong consistency and eventual consistency. And I want to explain this through the lens of the payment part of plant based pizza. And if it's close to a meal time for you now, I really apologize for this next few minutes because I want you to imagine that you've ordered your pizza and you've walked down to the pizza shop. You walk into the pizza store to pick your pizza up. You walk in, the smell hits you in the face. You can almost taste the pizza. You can see them being prepared behind the counter. You think you see your pizza being boxed up. You get super excited because it's almost time. But there's one thing you need to do before you can take that pizza away. You need to pay for your pizza. And you have two options at this point. The first option you have is to pay with cash. You can take out your purse or your wallet, take the money out, hand it over to the cashier, and the cashier will put that in the till. That's an example of a strongly consistent transaction. It's strongly consistent because you can absolutely guarantee at that moment in time that the money has changed hands, that transaction has happened. You open up your purse and there's no money in it. Alternatively, if you were to pay with your card, all you're doing when you pay for something with your credit card is that you're making a theoretical guarantee that at some point in the future the money is going to change hands. You might pay with your card, take your pizza, walk out, get back in your car, check your bank statement, and the pizza hasn't appeared on your bank statement. And now you're celebrating because you think you've got free pizza. Free pizza. That'll be wonderful. What you then realize is that eventually, of course, as you all know, that that transaction is going to come through eventually and it will eventually be displayed on your bank statement. This is an example of an eventually consistent transaction. Eventually, providing nothing else happens, all the different services in your system will converge on the same view of the world. But there's no way you can guarantee at any point in time that another service has seen a specific piece of data. And eventual consistency is one of the trade offs you make when you build eventor and systems. You can't solve for it, you simply need to embrace it and architect for it correctly. Because architecture is just balancing a whole bunch of different trade offs and this is one of the trade offs you make. Another thing that's really important with event driven systems is observability. When something goes wrong, how the heck do you work out what's gone wrong? And let's revisit one of them schema patterns from earlier. And I'm going to look at the metadata data version here. And what you'll notice I've done is introduced two additional properties to this structure, a trace ID and a span ID. I'm using open telemetry here and if you're not using open telemetry, this could be as simple as just a correlation ID. The important point is that you have a common piece of data that's passed through every single one of your services. And that way whether you're using distributed tracing or you're using structured logging, you can include the same piece of data and actually determine cause and effect. Cause and effect is really hard in event driven systems because producers just publish events and consumers maybe or maybe don't consume them. So you need to be able to really easily link these requests together. And doing this with trace IDs, span IDs or correlation IDs is a really helpful way of doing that. Another question I commonly get asked with event driven systems is what do I do at the boundaries of my system? Sometimes surely I'm going to need some synchronous communication. If you think about the front end to back end communication. If the back end is driven completely by events, what happens when my mobile client or my web client needs to actually communicate? And really you can't just have asynchronous communication everywhere. Somehow, somewhere you're going to need some kind of synchronous API and there are patterns that can help you here. I'm a really big fan of having these really thin APIs at the boundaries of a system. This API might perform a really small amount of validation when a request comes in, maybe stores some data in a database, and then actually publishes an event to an event bus. Commands are sent to your system, and then the crux of your business logic happens using events asynchronously. So now all of that meteor business logic, that can take a little bit longer to run, that can all be shifted and happen asynchronously. Now, I mentioned the word command and event there, so let's consider quickly the different types of messages that you might see, which are commands, events, and queries. Now the difference between a command and an event is mostly a statement of an intent, but they deliver a very similar outcome. I'm either asking a specific service to do some work, or I'm observing some functionality that has happened and I'm running some other functionality off the back of that. Commands are typically at the boundaries of your system, where it interacts with the external world API request coming in. Submit order aws an example events, then, at least in event of an architecture, are the first class citizens that drive the vast majority of your business logic. And then queries are mostly about reading data, reading a specific view of the world that's optimized for a specific use case for external clients to receive data, or your front end to display the status of your order as you furiously refresh the page, wondering where your pizza is. And it's important here to think that both commands and events can both run asynchronously. You can do asynchronous commands in an event driven system, much like you can asynchronous events. But let's pull all this together now to find out exactly how this can all work as events flow around your system. So imagine you get a submit order request that comes into your order API, your order API does some validation, does some work, and then it publishes an order confirmed event onto your event bus. At this point, the kitchen service is incredibly interested in that order confirmed event, and it grabs that to do some work. And you'll also notice that the order processing service is also interested in that order confirmed event, and it's using that to update a view of the world ready for any requests to come in from the front end. Once the kitchen has finished doing its work, it publishes an order prepared event, which the delivery service is then interested in to actually get the pizza and deliver it, but to the person who ordered it. Again, your order processing service is also interested in that order prepared event to update the status of the order in the order processing database. And then finally, when that get order status comes in when I'm sat there furiously refreshing my mobile app that can return really, really quickly because I've got this optimized view of the world created by my audit processing service. If this all sounds wonderful and exciting and magical, and you really want to start building an event driven system, but you don't just know how, what I'd encourage you to do is to start small. Don't try to make your entire system event driven overnight. Pick an area, get it right, and then start to scale up. And if you're working within an existing system, let's say you have the same scenario as plant based pizza and you've already got HTTP communication between all your microservices. One of the low impact things you can do to get started is to simply start adding events to your system. Even if you still prioritize HTTP communication, starting to publish events onto event buses gives you the potential to then add any new functionality using events and a more event driven architecture and get used to your systems working with messages and events. And then over time you can hopefully shift away from that more HTTP based communication. And remember that event driven architecture is fundamentally a communication pattern. It's a way of modeling the integration between your systems at a technical level. Of course it's about the technical integration. It's a way of integrating systems together, but using business events. But it's also a useful communication pattern at a business level. At a human to human level, one of the hardest things in software development is communication. The language you use to talk about your system between the technical and the business people can be one of the biggest challenges. If the technical people are talking about microservices and rest APIs, your business people are going to glaze over and probably stop listening to you. And remember, words and stories are the master key to human success. And given that, at least for the moment event, if software development is fundamentally a human endeavor, then this directly impacts the potential success of your system. Now, event driven architecture gives you that language to talk about your system across your entire organization, and it does that with some nice technical benefits as well. It gives you looser coupling, a more easily evolvable system that allows you to extend and change in a more flexible way. Of course, like everything, it isn't without its trade offs. Silver bullets do not exist in software. All you have is just a big collection of trade offs. And in this talk you've learned about some of these trade offs, some of the pitfalls, and hopefully some practical patterns that you can take away and use as you start building event driven systems. If you want to talk more, please feel free to reach out on any of the social media platforms. I have a lot of content on my YouTube channel talking about serverless and event driven architecture. I'd love to see you all there. And finally, thank you all for taking the time, but of your day to day to come and hear me talk. I'd really, really appreciate it. Enjoy the rest of the conference.