Conf42 Cloud Native 2021 - Online

Automation is where a project starts

Video size:

Abstract

Cloud native is all about speed, time from merging a PR to have this feature in production is priceless. The unique successful path is starting automation development immediately.

This strategy as the side effect of incrementally hardening the automation layer, testing in prod the automation layer.

Summary

  • Ricard Miranda talks about moving from monolith to services. Each service should be loosely coupled. It should be possible to develop, maintain and test and deploy independently each service. The amount of code that you need to develop for your application increases much more than the functional requirements.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hi, I am Ricard Miranda, and I'm here to talk to you about automation in modern project development. I'm a consultant at closer consulting. Let's start with movement from monolith to services. A monolith is a very big application, and we are now observing a movement from the monolith into splitting the monolith into several services, so that coordinate among themselves. What are the characteristics of a monolith? Well, a monolith usually is a very complex system with very complex code, very hard to understand and usually very hard to modify. In reality, the monoliths tend to evolve into a big ball of mud, where there is a very strong coupling among several parts of the system. So it is very hard to modify one part of the system without impacts in other starts of the system, even when these problems do not occur. Nevertheless, a new camera usually takes a long time to understand the monolith, can take years to really understand the system. The advantage of the monolith resides mainly in the easiness to communicate inside the monolith. A call to a function usually is very straightforward, so you don't need to worry about latency or the security of the communication among functions of the same project, or to discover where is the function. So very easy to communicate inside the monolith. When you go into the services world, you go into a view where each service is simple. It just does one thing and does that job well, so a newcomer will in a couple of days understand the service. It's also easily to disposable, so you can write a new version of your service and benchmark against the old version, and see if this new technology or new language or some other aspect is worthwhile to keep, or you just should maintain the old version. The complexity emerges at the coordination level, because you start to need to worry about contracts for the communication among services. You need to worry about discoverabilities. Security of the communication and the deployment of this system composed of services is more complex than the deployment of the monolith. There is also another very big difference between the monolith and the services. A monolith responds to a variations in the workload with scaling up and down. So when you want to increase the workload, you install the system in a bigger machine. And if you no longer need to reply to such a big workload, you decrease the machine. So can be very expensive, because if you need a really big machine, it can have a very big economic impact, but it may even not be possible to purchase a bigger machine. And it's also very slow this reply, because usually it takes a long time to install the system in a new machine. For instance, when you go to the services world, you go to a very different approach to the scalability of your system. So usually each service scales out when there is an increase in the demand. So it instantiates more units of the same service. So if you have a larger workload, you have more instance of the service. When the workload decreases, you scale in, so you kill instances of the service. It's much more economically, and as you can see, you don't need to scale all the services, just go the services that require the scaling out or in to have a better response to the variations of the workload. So what we need to do have to be able to answer to the problems of this increasing complexity of the services world. We have a wish list. Each service should be loosely coupled. It should be possible to develop, maintain and test and deploy independently each service. What we observe is that the amount of code that you need to develop for your application increases much more than the functional requirements you start to need to develop code for automation tasks in terms of CI, CD and testing and so on. That I will speak a little bit more later in this presentation. In terms of tooling you need, git manages all the files that you need in your project. Usually I take the approach of having a git repo for each service, but some people take another approach. I think this is the one that has been working the best for all my teams. The automation tool. I'm thinking about something like Jenkins, where you automate development tasks like building your artifacts, testing or deploying. Artifactory is a repository where you store versions of your artifacts. There is another very important tool that is the infrastructure as code. I'm thinking about tools like terraform, where you describe the infrastructure that you need for your application. Everything from virtual machine to databases and tables or vpns. Everything that you need should be described as code. How does it work? You have continuous integration to make new versions of your artifacts. Each time there is a merge of a branch to main in git. So every time you do a pull request and pull request is approved and then there is a merge automatically. A new version of the artifact should be published in the artifactory. And when you want to deploy an artifact in an environment, you just run a deploy script where you say I want this version in this environment and it automatically is deployed on the target environment without human intervention. And so how do we tackle the complexity of everything that I've been talking about as we have seen, we have to deal with much more than the fractional requirements because we need to build continuous integration layer, we need to describe everything as code, we need to build a continuous deployment layer, and we need to automation all kind of tests, much more than t unit tests. You need integration test, acceptance test, performance test, security test, everything that you really need to be able to put your application in production. And I want to stress the fact that there is no space for human intervention in terms of tasks. Humans can only be present as gatekeepers. For instance, you may require human authorization to deploy your application in production, but it's just a gatekeeper, it does not perform a real action. And in the end we have very good news, because the anti pattern of hands on my machine just disappears. You no longer run anything in your machine, your machine just needs an IDE, you don't need to run anything in your machine, don't need to do tests in your machine, nothing. You run pipelines. If there is a problem in the pipeline, there is a report that you need with the problems and you need to solve those problems. If the pipelines go green in the end of a process, you are sure that it is ready to production. And so automation is not an afterthought. As we have learned from test driven development. You need to do the tests before you write the code. So is it an iterative process that starts with writing tests, writing the code that solves the test and refactor. And this is a cycle, and you go forever in this cycle. During the time that you are improving your application, the same approach should be used for the automation. So you need to first have the infrastructure scode, you need to have the automation scripts that you need for your application at stage that you are. For instance, if you are really in the beginning, maybe you just need automation script and you start developing. And as your application evolves over time, you need to add the scripts for testing or for deployment. And same with the infrastructure as code, you may modify it, put more infrastructure in your project, et cetera. But it's always an iterative process. And this approach has two big advantages. In one end, if you build incrementally, you are sure that you don't leave anything behind and everything is querient. The same way that when you do TDD, you are sure that you achieve the level of coverage that you want and the code is testable. And it's easier this way because if you try to do the automation layer in the end of the project, the effort required is much bigger than with an incremental process. There is a second very important reason to do this approach is that you test the automation layer in prod so you gain muscle in your automation layer. You are confident when you reach production environment that everything is working fine. In terms of automation, your system is resilient, is more secure, is more error prone. And this is the reason why I say that automation is where your project starts. Thank you. It was a pleasure.
...

Ricardo Miranda

Big Data Consultant @ Closer Consulting

Ricardo Miranda's LinkedIn account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways