Are youll an SRE, a developer, a quality engineer who wants to tackle the challenge of improving reliability in your DevOps? You can enable your DevOps for reliability with chaos native. Create your free account at Chaos native. Litmus Cloud hello, my name is Rafael and my talk is titled five levels of high availability from multi instance to hybrid cloud. But first a few words about myself. I am the tech lead of the cloud native team at Hazelcast before I worked at Google and CERN. I'm also can author of the book Continuous Delivery with Docker and Jenkins and from time to time I do conference speaking and trainings, but on a daily basis. I'm an engineer, I'm a developer and I live in Krakow in Poland. A few words about Hazelcast Hazelcast is a distributed company. We're always distributed, always working remotely, and we produce distributed software. Our software is open source. What we do is, you may know Hazelcast, it's an in memory data store and stream processing platform. So people use it for machines, for processing everything where latency matters. So please check it out. Our agenda for today for this talk is pretty simple. So there will be a short introduction about high availability levels in general, about high availability in general. And then we will walk through all levels that you can use in your system while deploying, while designing your system. And while I'll be talking about these levels, I will always focus on the solution, but also on the trade off. So if you design your system to be available in let's say multiregion, what does it mean to you? What will you will achieve, but also what you will lose. So I will focus on the trade offs. So let's start with the introduction. So currently, any system, basically any system you want you build currently is based on services. So in the center, I mean everything now is a service. Sometimes we call it a microservice, sometimes we call it microservice even though it's huge. But it doesn't really matter. It doesn't matter if it's a microservice macro service or even a lambda function. What matters is that this part of the system with which you communicate is usually stateless. And stateless mean we can replicate it, we can deploy it wherever we want. So stateless thing contains the business logic and it's usually can easy part of your system. So just imagine for a moment that our system is just a bunch of stateless services, like a calculator service that can add two numbers. So then that is all your system. Then it's relatively simple, you can deploy it anywhere you want, it will work the same on my machine, on my laptop it will work the same on AWS. If you deploy it in 1 million replicas, it will still work the same. So usually the stateless part, the service is not a problem, but that is like usually the stateless service is connected to a bunch of the stateful things like a data store or a queue or it can be some other services, internal or external. So something that has state from our service perspective and this stateless, stateful things are the problem in our system. So the data, we can say that the data is the problem. So if there is no data, there is no problem in terms of high availability. So with this in mind, I will always focus on the data, but I will always give you the whole picture as well. So with this in mind we can start from the first high availability level, which is single instance. Single instance means no high availability at all, just look at the diagram. So single instance is kind of. If you deploy your database, can application everything on your laptop that is a single instance. Now why would you ever do it? I mean it's not practical but there are multiple things that becomes very easy when you deploy everything on a single instance. One thing is the latency. So I did a latency experiment in my scenario I deployed an application with embedded hazelcast. Hazelcast is just a data store, so it doesn't matter, it's hazelcast, it's just to show you what happens with the latency if you deploy everything on one machine so you can say like a service and a database. For us, database is hazelcast and I measured the latency of the retry write so it was 80 millisecond per 10,000 of entries. It doesn't tell us much, you can already see it's a very low latency but we'll compare it later to what happens with the latency as we go farther with our high availability levels. So if you deploy everything on a single instance, what it means to you, you have no high availability or no scalability but you have very low latency and you always have a data consistency because you cannot have consistent data if you just have one instance of your data store. And there are some valid scenarios for such a deployed even though it may seem crazy but imagine like let's say edempotent both job that you can always repeat there's no problem running this on a single instance. So there might be some cases where you deployed your system in that way, but okay, you didn't come here to listen to some development environment. So let's go straight ahead to multiinstance. So level one, multi instance. So multi instance means that if one machines is down, the system is still available. How can it look like, how will be the classic diagram for such a system? So a request goes to our system, it will usually go to a load balancer, which will forward the traffic to the application service replicas which are deployed on at least two machines. And this stateless part of the system uses some stateful part of the system. So in our case, let's look at some data store which is always also deployed on at least two instances on at least two machines. If you look at this diagram, there is only one thing that is single on this diagram and it's load balancer. However, load balancing is completely separate topic and it's somehow as a DevOps Persona. Usually this problem is kind of solved for you because nowadays these load balancers are super reliable and youll can just, with a few clicks you can have it done. So we will not focus on a load balancer so much. We'll also not focus on the stateless part because we already said that that is the easy part of our system. So we'll focus on the database. So in our scenario, the assumption here we have two database instances which are connected with a fast and reliable network. For example, you could start two EC, two instances on AWS. That is our scenario right now. So we have two data stores, two databases, and we would like to use them. So the problem, what we are trying to solve right now is called data replication. And at this point, let me give you a short history and a very simplified story, how this data application was approached and used. So the SQL databases, not so long time ago, it was the mainstream. Some may argue that it's still the mainstream, but with SQL databases, how this problem was usually approached was that we had like so called active passive or master slave replication, meaning that all application services, they use only one database instance, at least for the writes. So you write only to zones replica. And if anything wrong happens, then we just do a failover to the second replica. Because all the it, everything we stored in a machine one is replicated to machine two. If there is a crash of machine one, we do a failover so we don't lose any data and our machine two becomes the active replica. And that is how most replication looks like nowadays. You can combine it with sharding, no problem with that. Now with the movement of NoSQL and people thought like we need something better, this master slave replication. So it's maybe too difficult for developers and sometimes not that good. So most, like NoSQL databases, kind of abstract this replication way. They do so called clustering, which means that your application, they use all the instances of database, so they connect to each of the instance of the database, and each of the instance they stores the primary partition of the data and also the backups of some other partitions. We always store the backup of some partitions. So if one machine is down, no problem with that, we migrate the data and your application, they can still use you don't lose any data to go a little deeper into. And let's look at specific example, because actually each database solved this problem a little different. So it's good to focus on one example. So let's focus on hazelcast, how it looks like if we go a little deeper. So hazelcast is a key value data store. So we partition all the data by keys, and now each member stores some partitions of the data and backups of some other partitions. So the rule here is that the backup of the partition is always stored on another member than the partition itself. So for example, partition one is stored on member one, so its backup backup of the partition one will be stored on member two or three, but never on the member one. In our case, it's stored on the member two. Now, in any of the member crashes, at any point of time, you don't lose any data because the backups are promoted to the main partition and everything works. So from the client perspective, from application perspective, you may not even notice that, okay, we have like this. So this is more or less how this nosql tried to address this replication problem. But now the question is, should these backups, no matter if they are clustered or if they are really master slave, should they be done synchronously or asynchronously? In other words, should we favor consistency, so we do a backup synchronously so we are sure about the data? Or should we favor latency and do it asynchronously? And before I answer this question, I did another latency experiment, and this time I deployed hazelcast as a database on two machines and features the latency of read write again. And as you expect, it's way worse than a single instance because you cannot beat like one machine communication, so they will not beat it. Here you need to have like a network involved. But if you look at the number, it still doesn't look that bad. It's still under one millisecond per entry, which is usually acceptable. And this is with synchronous backups, so we are sure that you will not lose the data. So this is pretty good. That is why in most cases in NoSQL databases like by default, you will see that these backups are done synchronously, but you can obviously change it. Is it the same for this SQL and this more complex databases? Not necessarily. For example mysql, by default they do it asynchronously. So you favor latency here because otherwise the latency would suffer. So a short summary, what has been level one for you so you can achieve data consistency because you can do synchronous backup, it's still an acceptable latency. Most tools support it, so it's easy to set up. Most cloud specific toolkits support it. And even if you've done on premises like Hazelcut or any other tool, it's usually very simple to set it up from the downsides. If you would like to access this, you are still inside one region. So if you youll like to access the data from another region, then you will have high latency accessing from another regions. Okay, let's move forward and from multizone, from multi instance, one step up is multizone. Multizone means that if one availability zone is down, the system is still available. How does it look like on a diagram? So here is a diagram for multizone. So again, request goes to cloud balancer, load balancers forwards to the traffic, to the application services, they use databases and now the application services, they need to be deployed in at least two availability zones. And the same with the databases, they need to be deployed in at least two availability zones. So if you look at this diagram, it's pretty similar to what we've seen with multi instance. That is why the first question you might ask is is it any different than deploying to multi instance multizone? Does it make any difference or just do the same? I just change my configuration file and that's it. So I would say it's not much different, but there is some difference. So let me first explain why it's not different. So I did another latency experiment and this time I deployed four hazelcast instances, two in each availability zones, and I measured the latency. And obviously it's worse than multiinstance, but it's not way worse. It's not even twice worse and it's still under one millisecond per entry. So that is actually usually good enough. That is why you can just move your deployed to multizone and do it with the same. But I also mentioned that there is a difference, there is some difference between multiinstance and multizone. And to understand the difference, let's focus on one example. So this is exactly the scenario I used for experiment. I deployed four hazelcast instances, two instances in each availability zone. The assumption here are that the network between availability zone is fast and reliable because that's how cloud providers work. And so if you would like to reproduce such a scenario, you just deploy four ec, two instances in one availability zone to another availability zone. And that is your scenario. You remember this diagram, probably from a moment ago I presented this. So we have like partitions and the backups, and the rule is the backup is always on another member, so it always works good. But imagine what would happen if we store the partition on the member one and the backup of the partition on member two. However, it happens that machine one and the machine two, they are in the same availability zone. So now if zone one crashes, what happens? We lose the data because we lost both the primary partition and the partition backup. That is why the same deployed we moved it from multiinstance to multi zone and it doesn't work anymore. We thought how to solve it in hazelcast this problem and we came up with the idea that a feature called zone aware, so we groups the partitions by zones and this is actually how we enable this in the configuration. And if you enable this, then hazelcast will know in which higher verbatim zone it is located and it will keep the backups always of the partitions always in a different availability zone. So that if zones zone is down you never lose the data, since the concept of the availability zones is related to the cloud. So it's different in AWS, different in azure, different in GCP and in kubernetes. In general we provided plugins for each environment. And by the way, these plugins are also used for the discovery and they used the given API, for example AWS API, to check okay, in which zone I am located. And then Hazelcast knows where to distribute the partition backups so that in result you have the partition backup always in another availability zone. And if one zone is down, you will never lose data. So that is why I said there's not much difference in terms of latency and in terms of the configuration. But please make sure that your data store is zone aware or supports like zone aware feature and don't take it for granted that it works this way. Short summary so level two multizone, I would say it's currently top one choice because you have very highly available system youll can keep the data consistency with synchronous backups and a lot of cloud specific toolkit that supported. Now please make sure that your tool really supports zones aware. So read how it works. You need to go a little deeper. And from another downside is the same as multiinstance. So we are still in the one region, meaning if we deploy our system in Europe and someone from us accesses our system, then latency is high. And that is the problem we will try to solve right now, going from multi zone to multi region. So multi region will be. Yeah, it's multi region means that if the whole geo region is down, our system is still available how it looks like on the diagram. So that will be the diagram for us. So usually a request goes to Geolo balancer, and it's forwarded to the regions which is the closest to our client, and then it's forwarded to one of the application services, and they use some data. And between the data stores in different regions, we have something called geore application. So the assumption here we have our data stores deployed in different regions, and the network may be slow and unreliable. For example, if youll would like to simulate, you can deploy two ec, two instances, one in Europe, one in us. On each of them, start your database, and that is your deployed. You may think, but what's the problem of regions? Like, why multiregion needs something like georeplication, why is it different? So the problem is the distance. So the physical distance is something that is actually the problem. If you look at the map, you look, for example, two regions, Finland and Oregon, it's like 10,000 between. So even if we calculate, like take the speed of light in a vacuum and take the distance, so the round trip time of light in a vacuum would be 60 milliseconds. So it's already a lot. If it's not a vacuum, but a fiber, it will be at least three times more. And it does not takes any routers root and anything, it's just light. So this is something that we cannot solve. That is why we cannot use just a standard replication in between our data stores. We need to use something called georeplication. So georeplication is just like, you have two data stores, and you would like to have the same data in both of them. So youll need to replicate them and it needs to be done asynchronously because of the physical distance, your latency would suffer too much. If you don't do it asynchronously, youll data store must support it, and you must be prepared for the data loss, because you use asynchronous communication and there in general like two nodes, active passive or active active. So active passive is like you write only to one data store and the second one is not used, or it's used only for reads. And with such a scenario you need to be prepared for the data loss because maybe you wrote something to the active cluster and it crashes before replicating the data. So you need to be prepared also for eventual consistency. And in the active active mode means that you can write to both clusters. And then you need to be prepared for the data loss, for eventual consistency, but also for the conflict resolution. Because what happens if you modify the same data at the same time on two database clusters? You need to have a strategy how to solve this conflict. So in most cases, the simplest strategy is, for example, the last one wins. But you have to think about it inside your application logic, because your business logic needs to be aware what to do if you have a conflicting data. That is why it's actually very complex to say to reason about active, active georeplication. In headlocks we call this georeplication one replication. And this is actually how youll set it up. So it's actually very simple. You set up the one replication and the IP or the hostname of the target cluster and that's it. It will replicate the data. The only thing with this jury replication, it does not sound easy. So do I really need to lose the consistency if I would like to go multiregion? And I did another latency experiment, and this time I did something I should not do. So I've set up one hazelcast cluster with synchronous backup over two geographical regions. And I measured the latency and that was the result. So you see it's way higher than multizone and it's usually at the range. That is not acceptable. That is why you will never use synchronous replication across multiple geographical regions. And I did the same with the georeplication, obviously it was synthesis. The replication is asynchronous, the latency is the same as multiregion. Now if you look at this chart, that is the reason why you will never see a Kubernetes cluster, which is across multiple geographical regions. You will see a Kubernetes cluster across multiple zones, no problem with that. But you will never see a Kubernetes cluster across multiple geographical regions. And the reason for that is that internally Kubernetes, cases etCD, which is data store, which is consistent. So you cannot have this eTCD across multiple regions, because it will just be super slow. So technically you cloud have a Kubernetes cluster across multiple regions, but it will be so slow that you will not be able to use it. Yeah. So you will see for kubernetes some other options, like federations, but you cannot have one consistent cluster across multiple regions. Short summary of the level three multiregion. What does it mean to you? So we are super high available. You can achieve low latency when accessed from multiple region. That is great. And sometimes you can use some cloud specific tools, sometimes not. But on the downsides, you need to use georeplication, which is asynchronous. And you need to be prepared for the conflict resolution, eventual consistency. So it's way more complex to reason about a system like that. Okay, from multiregion we go to multicloud. So multicloud means that if zones cloud provider is down, the system is still available how it looks like on the diagram. So again, you will have some load balancer, which load balances the traffic, and you have basically the clone of the system on a different cloud providers and a replication in between. And this replication will be asynchronous, like a geore replication. So you may ask like what's different from multiregion? This diagram is actually like the same before. It's just you have multiple cloud providers, but that's it. But there are some differences. So if you deploy the same system on different cloud providers, you cannot use cloud portfolio and you cannot use VPC peering, and you increase the cost because you need to maintain the same infrastructure and the same system on a multiple cloud providers, which is more complex and costs more money. But is high availability the only reason for multicloud? And obviously no, it's not even the main reason. The main reasons for going multicloud is like avoiding Vernon lock in. So you don't want to bind yourself to one cloud provider. Because for example, imagine you have the same system deployed in different cloud providers. That is way better for any cost optimization, for any cloud cost negotiation, because you can go to AWS saying, okay, by the way, I'm also deployed on Azure and GCP, so can you lower the bill? And that is what happens actually with cloud providers. You negotiate with them if you are big enough. And it's also about risk mitigation if youll don't want to be bound to one cloud provider for some business reasons. Also, sometimes it's also about low latency. When we've built hazelcast cloud, hazelcast cloud is hazelcast AWS a service, and we build it on top of different cloud providers, because our users are also deployed on different cloud providers and they would like to have the lowest latency possible. So that is why we are built on different cloud providers. And also some data protection regulations or sometimes you build like heterogeneous environment when you use different cloud portfolio for different parts of the system. If you go multicloud, what does it mean to you? So you don't have vendor lock in, which is very good. You are in better position with the cloud cost negotiation, and you can have low latency of access from multiple cloud, but the setup is way more complex and you cannot use cloud portfolio if you really have the clone of the same environment on different cloud providers. Okay, and the last level for today is hybrid cloud. So hybrid cloud means if all cloud providers are down, your system will still be available. But really, is it possible that all cloud providers are down at the same time? No, it's actually not even possible. It's not even possible that one cloud provider is down. I think it never happened that the whole cloud provider, all services were down. There are different reasons for hybrid cloud. So then why would you use hybrid cloud? So it's usually about data requirements, regulations, data security. Sometimes there are regulations, for example, that the data of your bank customers needs to be stored in this country or in this building or in this city. So then part of your infrastructure is on premises, on your own servers. The other use case is that you are moving to the cloud. You had your own infrastructure, but you're moving to the cloud. And as a process, you build a hybrid cloud environment. And it's also sometimes about cost reduction. If you are big enough, then cloud may not be the cheapest thing you can get. How does it look like on a diagram? So again, youll have load balancer and the same system deployed on different cloud providers, but also the same system deployed on your own machines. So at this point, actually at this point, but also at the point of multicloud, you will usually need some abstraction. And this abstraction because it's just not manageable. When you manage different servers, different machines, it's just too difficult, too complex at this point. So usually what people do, they have some abstraction layer and they interact with this abstraction layer. And so you install kubernetes. Nowadays this abstraction layer is usually kubernetes or some derivatives from kubernetes like Openshift or IBM cloud. And you deployed this kubernetes on each environment, on the AWS, on GCP, and on your own machines. So at least you have the same interface to interact with. And it's actually how most companies now build multi cloud and hybrid cloud systems. Short summary so hybrid cloud means that you have no cloud lock in. You can have achieve low latency accessed from any custom network because you are the master of your infrastructure. But the setup is very complex and it usually requires this kubernetes or openshift layer. And it costs a fortune. So as a short summary, a short summary. What I propose is not to repeat much what I've said, because that could be boring, but to at least some, to summarize basically what I've said. So I propose like a graph how I see it. Like comparing complexity and cost to the high availability level. So this is how I see it. So at the beginning, single instance is very. It's not complex at all. But then if you go from single multiinstance to multi instance, it's actually a big step. Because you go from nondistributed system to a distributed system. So you need to think about data, applications and all those things we mentioned. If you go from multi instance to multizone, actually it's not such a big step in complexity. In terms of complexity, you need to make sure that your database is zone aware. Then from multizone to multiregion, it's another big step because you have long distance. So you need to be prepared for georetication. You cannot have synchronous single point of data going from multiregion to multicloud. You cannot use cloud tools. And from multicloud to hybrid cloud is another big step. Because you need to maintain your own infrastructures, which high availability level is for me. So I would say if youll don't know, then this is your level. This is what I call top one choice. Because multizone youll are very high available. But you can still use like synchronous data store, which really simplifies your reasoning about the system. If you are a software cabo, you can use single instance. Multiregion is if you need to go worldwide with your system. So you would like to achieve low latency from different regions. And multicloud and hybrid cloud is basically for the like, for the reason of compliance and security. So basically banking industry. So youll need to have a reason to do it. And with this last slide, I would like to thank you for listening to this talk. It was a pleasure speaking at Conf 42 conference. Enjoy the rest of the conference.