You event driven architectures problem do they actually solve? Hi and welcome to Conf 42. I'm Bogdan Sucaciu and together we'll demystify event driven architectures with Apache Kafka. I've been working with event driven architectures a couple of years now. Currently I'm a tech lead at Deloitte Digital, a digital consulting firm. And probably some of you may know me from plural website where I teach about various subjects, event driven systems being one of them. Speaking about event driven, this is the subject that we're going to be focusing on today. My goal is rather simple, to change your perspective about it. For a long time they have been marketed as the solution for real time or near real time systems. But is it true though? I mean, we already have real time systems that are not based on events. We create a new user, we make can order, and so on. It's not like I'm going to create a new user, go and grab a coffee, and then 5 minutes later I can come back and finish my order. In fact, only a couple of use cases really require event architectures, and most of them are related to real time analytics, fraud detection, sensor readings and so on. But still, event driven architectures, something else to the table, something much more powerful than you expected. And to illustrate all of this, I'm going to use Apache Kafka. Actually, I'm not going to be using on Kafka that much. If you've already used Kafka, there is quite a big chance that you won't learn anything new today. But still, Kafka is an important piece of technology in the event driven world. Why? Well, I'm going to drop some clues during the session regarding why Kafka makes such a great option for event driven architectures. Without further delay, let's get started. And what better way to start with than one of the most powerful architecture patterns? Demonolit. Yeah, you heard me right. Demonolit is indeed an extremely powerful pattern due to its simplicity. Basically, we have a bunch of lines of code, and all that code is running on a single process. Also, we often have a database attached to it in order to store various data. Most of the time is some sort of a relational database. However, it has quite a major drawback, scalability. And I'm not talking just from an operations perspective like virtually scaling a process, but I'm also talking from a team perspective. I think we've all been there endless conversations on collaboration apps about basically changing the color of a button to pink. So working with monolith is not really scalable or fun. So how can we fix this problem? Well, some very smart people came with the idea, what if we take this big application and split it into multiple services? It first started with SoA service oriented architecture, but it was quite tricky to determine the right bounds for the applications. So this tendency naturally evolved into what we call today microservices. Each microservice gets a different piece of the business domain. This way separate teams can work on separate applications, allowing to scale down or up the number of teams working on these product. Exactly the same concept can be applied to infrastructure. We can scale down or up the number of instances only for the applications we need to. However, there is a slight concerning issue. As we all know, in software engineering we don't have good or bad, right or wrong, we have tradeoffs. So by using microservices, we now have to solve a different set of problems related to data access. A lot of times microservices sound very utopic, like each microservice can evolve independently without having the need to coordinate with another microservice. But I think we've all been there. Practice doesn't transpose 100% from theory. Most of the time we're using synchronous APIs such as rest or RPC calls to exchange data between our services. Let's say that one of our users makes a request to one of our services. From there we have an entire cascade of other calls that need to be done from one single rest call. We may end up with 20 or even 30 other calls. This happens because even if each microservice owns a single piece of the business domain, we will always need data from other domains to get the full picture. Let's say we want to make a payment transaction. It's not like we only need the credit card details. No, we also need the name of our user or even his address. And that is usually stored by a different microservice. So in order to get the full picture, we always need to somehow compose it by calling different microservices from our architectures. Sounds easy, but by doing this we are diving into a new set of challenges, such as distributed transactions, timeouts, dependency management, coupling or even data ownership. These challenges and many more others have been introduced by microservices and by the fact that we are now using distributed systems to perform our tasks. And that's what event event driven architectures really good at. They can easily solve distributed systems problems by introducing new concepts and patterns. But first, what exactly is an event driven architectures this is these main talking point of my presentation and I haven't even introduced it yet. Well, I was curious too, so I went to Wikipedia and I found this event. Event driven architectures shortly said EDA is a software architecture paradigm promoting production, consumption of and reaction of events. There are some interesting words being thrown in there, production, consumption, reaction and so on. So let's find out what they actually mean. To do that, we need to go back to our microservices to keep things simple. I'm going to pick only four services, blue, gray, purple and red. To make things interesting. The blue service is using reSt APIs to communicate with the purple and red services, and an RPC call to communicate with the gray service. Rest APIs are based on a request response type of communication, whereas with RPC calls we don't always get back a response. This flow can also be represented in another way using unidirectional flow of data. Now the blue service calls the gray service and the flow ends. But when the blue service calls the purples and the red services, the flow continues and they will provide some sort of data back to the blue service. This is interesting, but we still haven't solved any of the problems mentioned earlier. To solve them, we need another piece of the puzzle called event stream. The trick is quite simple. We now have two types of applications, producers and consumers. Producers send data to an event streams and consumers retrieve it from the same event stream. What's interesting is the fact that an application can be both a producer and a consumer at the same time. Just like the purple and the red services, these consume data from the blue event stream and produce it into another. Now, if we isolate only the producers, we can immediately notice that there is only one producer type per event stream. The blue service produces on the blue event stream, the purple service produces on the purple event stream, and so on. You should definitely keep this in mind when you're designing your event driven architectures. You could, for example, use multiple producer types to produce to one event stream, but that will complicate a lot. The consuming process. Then if we take a look over the consuming part, we can notice that multiple consumers can retrieve data from the same event stream. Also, one consumer is allowed to consume data from multiple event streams. So consumers are much more flexible than producers in terms of connection to different event streams. And by introducing only one concept, we actually tackled multiple issues. First of all, our applications are no longer coupled since the event stream is meant to serve as a decoupling factor and then data ownership is already simplified. Each event stream stores one type of data and any interesting party can easily consume it. But what exactly is an event stream? Well, an event stream can be represented by many things, such as logs, topics, queues, websockets or even reactive APIs. Now I know that some of you may be thinking, hey, how can a websocket be considered an event stream? Well, if you come to think of it, websockets or even reactive APIs are a bit more special scenario. The event stream and a producing application have merged together into a single entity. But if you think about the previous deductions, we still have only one producing application, and multiple consumers can connect to the same event stream. The only difference between a reactive API and a topic is that the event stream and a producing application are two separate entities. In fact, anything that you can asynchronously subscribe to can be can event stream. Any tool, library or protocol that can produce data in an asynchronous manner can be considered an event stream. These are, however, some fundamental differences between these event streams types. For example, queues, websockets and reactive APIs treat event differently compared to topics and logs. While using the first category, we can only react to events, we consume them and then they are gone with no option of getting them back. Topics and logs, on the other hand, can persist those events and we can replay them later on. This pattern is a bit more powerful and we'll see why a bit later. An important thing to remember is that kafka uses topics as event streams, and under the hood it uses logs to persist events. This is one of the main reasons Kafka is such can excellent choice for building event driven applications. Of course, Kafka is not the only system that allows this. These are plenty of other examples such as AWS kinesis or Apache Pulser, which work in a similar fashion. But Kafka has another ace under its sleeve, something that kinesis and pulser aren't that good at, which we'll see in just a few minutes. So in Kafka's world, event streams are represented by Kafka topics residing on Kafka brokers producers connect to the Kafka broker in order to produce data, and consumers have to do the same thing in order to consume it. This brings some implications to our way. We exchange data throughout the system. Firstly, we're now dealing with something called inversion of control, and it's quite a simple pattern. While using the classic rest API approach, it is the job of the producer to send these data to the consumer. Now, the producer doesn't care anymore. Its only job is to produce data to an event stream. Actually, it's the consumer's job to retrieve the data by subscribing to the event stream. So we are inverting the control from the producer to the consumer. By empowering the consumer to retrieve the data that it is interested in, then we have can out behavior. Multiple consumers can consume data produced by only one producer. Talking about producers, it is recommended to have only one producer type per event stream. Just to make things clear, I'm referring to the producer type and not producer instance. Multiple producer instances can produce to the same event stream as long as they are on the same type. And finally, producers don't really know consumers. From an application perspective, a producer is like hey, my only job is to produce data to this event stream, but I don't know who is going to consume it and frankly, I don't even care. Consumers have the same mindset. These don't know who is producing this data, they only care about their connection to the event stream. However, if we zoom out a bit from an architecture perspective, consumers and producers are well known in order to get the full picture of the business flow. So going back to microservices, our main challenge was data access. By adopting an event driven architectures, we're actually solving it. Our data is now really easy to access. We just have to make our consumers subscribe to an event stream and that's it. It's much easier to subscribe to one or more event streams rather than making 20 or 30 API calls. But still, just as I mentioned previously, in software engineering, we don't have right or wrong, we have tradeoffs. And the tradeoff for event architectures is consistency. While using synchronous APIs we have that strong consistency, and we're always sure that we're getting the latest version of our data in event driven architectures no longer have that. Instead, we have something called eventual consistency. In fact, the key to building successful event driven architecture is understanding and accepting eventual consistency. We can't just simply take a synchronous rest API and then transpose it in an event driven way. New patterns require new ways of thinking, so such attempts may fail due to the mismatch between these patterns. In fact, not all flows are meant to be event driven. In the real world, we will always find a mix between synchronous APIs and eventing patterns. Now, I've babbled a lot about events, but I haven't really said what an event really is. Well, to do that we need to have a look of our communication types between systems and the best way to start with is a message. A message is just some data being exchanged between two different services a and b. A message has a form, a body, but it doesn't say how the data is being transferred. That's why we need some more concrete definitions. The types of messages can be determined based on when these action resulted from the data transfer is happening. The first one is command a. Command will always happen sometime in the future. Even if we initiate the action now, the result of it will be perceived sometime in the future. Also, a command is a directed instruction, meaning that we know exactly who we are communicating with. An example will be rest calls service a, calls service b. A very important thing to know is that the fact that we don't always receive a response while using the command pattern. Sometimes we do, sometimes we don't. The opposite of a command is the query. Now, our service does not give instructions to another, but it actually requests some data, just like trying to request a database or a search engine. Queries happen in the present because we are requesting the current state of our data. Also, one important fact is that we are always getting a response back. Finally, we have events. Events are a bit different compared to the other two. They represent actions that have happened or have been triggered sometime in the past. As you've probably noticed, commands and queries require point to point communication service a, query service b, while events on the other hand are undirected, meaning that anyone can consume events without the producer knowing. Also can event producer will never receive a response from its consumers. In fact, there are two types of events that a producer can send. The first one is called event notification. And just like his name is suggesting the event producer is informing its consumers that an action has happened. I can actually give you a great example of this pattern in practice. I'm not really sure if I can mention their name, but let's say that one of the biggest ecommerce companies has an extremely interesting use case on their homepage. They are displaying some random items that you may or may not be interesting in buying them. However, at the bottom of the page they also have a questions section. Do you know how they are compiling those suggestions? Well, we tend to move the cursor based on the direction of our eyesight. If we look to the right, we also move the cursor to the right. So if we see something that may attract us on that page, we tend to hover these cursor over that specific item. The moment we do that can event is being sent to their questions service notifying that we have hovered over that product. So as we scroll down the page we probably hover over a couple of items. Then at the bottom of the page we are getting suggestions based on which items we have hovered over. Sounds really cool, right? This is actually one of the best examples of the event notification pattern because it really denotes that eventing aspect. The producer isn't interested in getting back a response, and it also doesn't care about its consumers. All the producer does it cares about is to send hover events to the backend system. So how would something like this would look in Apache Kafka? Well, we always have at least two applications, a producer and at least one consumer. The first step is to define the event stream, in our case a Kafka topic. By the way, I'm using Java here, but there are kafka clients that would work with many other programming languages like Golang, Python and so on. Then in the next section we have to define some configuration. Both our producer and consumer need to know how to connect to the Kafka cluster and what serialization format should use. Next we have to initialize a Kafka producer and a Kafka consumer. Pretty straightforward so far. Now things start to divert a bit. The producer has to create a producer record. You can consider a record as an equivalent to an event. It's just that record is the official naming in Apache Kafka. A quick fact about records, a record is composed of a key and a value. Keys are used to identify events and values store the payload of that event. On the other hand, these consumer has to subscribe to the same topic the producer is producing too. We can actually pass a collection of topics meaning that our consumers can subscribe to multiple event streams at the same time. Finally, these only thing left to do for our producer is to send that event to the Kafka cluster. I've talked earlier about inversion of control and how it's the consumer job to actually make sure it retrieves the data well. The way this works in Kafka is by using these poll pattern. The consumer will actively query these Kafka cluster in order to find if there are new events to be consumed. As soon as the poll method returns, we are free to do whatever we want with those events. However, the poll action only happens once. That's why we need to wrap it in a while. True loop the end result sounds like a broken record. Hey, do you have new events for me? Do you have new events for me? Do you have new events for me? If the broker has new events in the event stream, it will pass them further to the consumer. Let's actually see how these code would look in action. So I have four terminal windows. On the left I will be running a single producer. On the right I will be running not one but three different consumers. All three will do exactly these same thing. They will lock the consumed event. In practice, these would actually consume the events in different ways. Starting with a producer, we can see that it's producing messages containing an incremental number as the key and a random uid as the value. I've used a while loop to make sure that an event is generated every 1 second. Now let's start the consumers. Firstly consumer one, then consumer two, and finally consumer three. As you can notice, all three of them are receiving the same events almost at the exact same time. Just try to think about how much code it would take to write an application that does the same thing using synchronous APIs. Of course we would get some added benefits like strong consistency, but if we don't really need them, it's fine. The next type of event is event carried state transfer. I know it sounds quite fancy, but I think it will be quite easy to explain it using an example. Event carried state transfer is all about the changes that occur in the current state. Let's take for example, the address changed event. A user changes his or her address through a user interface. This is a change in the state. Now our producer needs to send the address changed event to the event stream. But how much information should that event contain? It should contain only the new address, the old one and the new one. What about the user information? Let's say we want to go minimal and pick only the new address. That event is produced to Kafka, which is then picked up by the consumers. Only then we realize that one of the consuming parties is a tax service that requires the oral address as well. Uhoh, what can we do in this case? Well, that service can query back to the source system and get the missing data. Do you notice the problem? Because we haven't passed enough information into our event, we have now introduced coupling between our services. That's why when you're designing your event model, take into consideration that all consuming parties should avoid querying back the source system. One other problem that we frequently encounter while using microservices is the distributed source of truth. Our data doesn't reside in a single place, but it is distributed across multiple data stores. So in order to compute the results, we actually need to query multiple databases, and it's quite tricky to do it the right way. Sometimes we may be tempted to apply some workarounds, like copying parts of the data from one database to another. But this also poses some challenges. How can we make sure the data is in sync? How can we make sure that the moment we compile our result, we have the latest version of the data? A lot of effort would have to go into writing, deploying and maintaining some database synchronizers. We are now working with distributed systems, so we need another way to tackle this, a more efficient one. And these is these event sourcing comes into play event sourcing is probably one of the most powerful patterns, but it is also one of the most demanding. It all comes down to an event log. You can think of an event log as a ledger where we keep various actions that have happened. For example, let's think of a bank account. When we open that bank account, we start with the initial amount zero. Then money is being deposited into that account. First we have a transaction of 500 and these another transaction of 200. Finally, we have to pay for that new tv and $300 are subtracted from that account. So event sourcing works by always appending new records to the log instead of updating an existing row in a relational database. If we want to compute the current amount, the only thing we have to do is to replay the log and go through all the transactions. By doing so, we can easily obtain the current balance of our bank account, which in our case would be $400. These is how event sourcing works. We always build the current state by replaying all the events from the log. You may think that these pattern is rather new, but we're actually using it for quite some time now. To give you an example, Git is actually an event sourcing system. We always retrieve the latest version of our code base by replaying commits from the log. So why is this pattern so powerful? Well, it solves our data access problem. Let's say that we have three applications that need some data. In the Kafka world, these would be some simple consumers that subscribe to a Kafka topic. Just a quick reminder. Kafka uses log files to store events. That's why it is such a powerful option when it comes to event driven architectures. Cool. Now let's say that two events are produced to the event log. The only thing the consumers would have to do is to consume those two events. They can be stored either in memory or any persistence layer. By doing so, we are building a socalled materialized view. We are building a current state based on events. If a new event is being produced, the consumers will pick it up and update their materialized views. But this doesn't happen in an instant. There's a slight delay between actually appending the event in the log and updating these materialized view. That's why we have to accept eventual consistency. We know that at some point in time our data will be consistent, but we don't know exactly when. Now, not everything goes perfect in production and things may fail. Let's say that one of our data stores goes down and we lose all our data. But that is totally fine. Why? Because all the other applications are not going to be impacted. If new events are being produced, the other applications can consume them in their own rhythm and update their materialized views. But what about the application that went down? Well, we can simply bring back an empty data store. Kafka consumers also have the ability to rewind and consume events from the past. That means we can start consuming from the beginning of these event log and recreate the current state. I always like to end something on a positive note. So what I want to say is that the game is on. We are actually witnessing a transition period and we are all part of it. A couple of years ago microservices were really popular and everyone was talking about them. Right now the same thing happens with event driven system we are changing our systems to adapt to a new mindset and event event event driven architectures. Step of all of this I would like to thank you for joining this session and don't forget I'm available to answer any questions that you may have. Feel free to connect with me on Twitter or LinkedIn. I'm always open for interesting discussion. Thank you again and I hope I'll see you next time. Have fun.