Hi. Hello and welcome to the case study of building operations team at Makolab. For last three years I effectively and efficiently built area of DevOps operations, maintenancesupport and support that delivers outcomes by going beyond the standard DevOps framework with diverse competencies. Every day I'm going out of the box. Extending classic DevOps approach in multidimensional operations that boost up projects and deliver highest quality. I'm from Wooch. Woodch is a very colorful city, the center of Poland and in case that you will visit it, let me know. I can guide you through the main areas that you should visit. Okay. My name is Grzegorz Sztandera and for the last 20 years I'm working for the banking and it industry. Twelve years for the project management and also last few years, as I said, building the DevOps area at Macula, providing services for multiple industries. Personally I like traveling to extraordinary destinations like on the picture you can see me in front of the fourth reactor of nuclear plant of Chernobyl in Ukraine and also I'm working on the free shifts because I'm working at Makolab and also I have two sons being creative means. Also that I'm a painter for the board game Minifix and recently I wrote a book stories for kids which is in printing process right now. What is the pipeline for our today presentation? First I would like to highlight our areas of expertise with our plugin approach of the operations team. So you will see what we reached after two three years of building that area. Then I will explain more about operations team and I will cover multiple stages and aspects of building this team for the last few years. Before the end and before the summary, I will present few projects that we were taking part of on a general layer and after that I will share a few takeaways with you. The most important from my point of view, which was crucial for this success story. Okay, let's start DevOps up. Plugin approach what does it mean? We are in the middle of the ecosystem, so generally I will explain the plugin approach later. But we are between the project developers, QAS and project management and we are taking care of everything. What is needed for the project does mean we are taking care of test int e to e environments and also supporting testing for the project. From the second side we are supporting customer, third party companies and end users taking care of everything around us. 360 we are taking care of environments including production, monitoring of the environments. We are providing the whole scope of it services and maintenancesupport building and delivering the CICD processes also hypercur with extended productivity and predictive maintenancesupport everything in the 24/7 manner of on call and if possible and if we will have time, we are providing also a lot of automation inside our work. I can divide my team services into three areas, as I said, Dev, Ops and sub. So Dev, it's strictly connected with coding and creating something. Ops in my understanding means that we are operating what we did create or what we receive to operate from the customer with extended productivity, predictive maintenance and so on. SAP refers mostly to itil processes and standard support extended with deep analysis of case within our team from my perspective as an operations manager, we are capable of creating any architecture and providing the ideas for service architecture for the customer to suit the customer needs. After that we are managing it in example on production and after delivering or having the application or the product that is needed to be on production, we can optimize and provide service optimization based on our know how and our operations. For example from the monitoring systems plugin approach, the sample activity for plugin approach is just in case of need. We are capable not only to provide the full scope of DevOps operations, maintenance and support services, but in case of need we can plug in into the project or into the customer environment, perform the tasks and then switch off the sample of that is for example building the automation with terraform for AWS environments. We had a project where customer asked us to look into the manually created AWS environment for the product to optimize it and after that to create it using terraform. So we were plugged into the project, we provided the terraform scripts, run it, deployed a new infrastructure and after that we switched off from that specific project. This is the plugin approach so we are not only providing the complex services but in case of need. Like a wanting army, we can act and provide the specific tasks that are needed by the project or the customer. On the screen you can see just the sample list of the activities that we can provide. Also, we are multi cloud team, so we are capable of providing the solutions on for example AWS. On azure environment we are the can in black, so we are available and we are providing the services during the day and the night. In case of needs. During the day we are providing knowledge sharing with the project or we are participating in the standard activities that project is having in the scrum or agile methodology. So to sum up, we are able to deliver the whole DevOps classic approach, including development and QA based on other Makolab teams. But we are involved in any of those eight activities, as you can see on the screen, in a different way of working in a different approach. Now we will move to the building of my team. When I came to Makolab three years ago, I had a team of three persons and after a month two of those free persons gives the resignation letter. So at the very first month of my potential of leading the team, I left with one person having one project on production and supporting in the full 24/7 manner. So it was not the best start. But as you can see on the chart, based on the FT number of employees in the team, it was rapidly growing. Between 2021 and 2022 I will focus on those two first years of building the team because right now it is quite stable and the competition are growing and the projects are changing so very fast. I was forced to deliver additional engineers into the team. It was done by adding a person by the both leasing, not by direct hiring, but generally starting from 2021, we are permanently, almost permanently looking for the new engineers to join the team. What brings it also so having 24/7 on production gives the necessary on being on the permanent on call for the first year of my existence in the team, supporting the team for providing the highest quality of our services based on the ITIL methodology. So in the first year I did over 100 overtime hours per quarter. And also the team as it was growing and the number of competencies both combine with the number of work that has to be done was rapidly growing. We had in the first year around 1000 overtime hours. In 2021, at the end of 2022, we had ten people on board. So from one to ten within two years, it was also combined with growing the competencies, growing the number of projects which I will show you later, I was recruiting mostly junior and regular DevOps engineers, each in different specialty. While to be able to do all of the possible tasks, as you can see on the screen, there are some amount of daily tasks common for the whole team. But case on the individual specialized into different areas. Like one person was more an expert in the databases, the second person was more the expert in IT processes. I was able to very fastly deliver the high quality for the first project and other projects that was started within the operations team at Makolab. What is the advantages of having the junior regular team on board with some areas of expertise they learned from each other. So based on the person that is specialized in databases, he didn't have the well and deep knowledge about IT methodology. So the person that is not knowing much about databases is an expert for him because he is an expert for the IT processes and opposite, the person that was taking care of the IT processes was saying that the database person is the expert for him. So the team was and is constantly learning from each other and they are experts for each other and they are not in the point that they do not see any possibility to grow within the team and within the company. Okay. I also have line manager who is taking care of the HR processes. And this at the very beginning stage of my journey of building the operations team at Makolab was very important to have such support. And also today having line manager in the scope of the team is giving a boost for both technical and business area. So generally the team evolved into having competencies in sub areas and having the commitment for the common tasks and providing the proper quality. Also, what I want to highlight that at the very beginning of creating this team, and also even today I'm looking on the personality even more than the technical skills, because I believe that we can learn everything, but we are not able to change our attitude as fast as we can learn the new technologies. So that was the time of building the team. It was quite interesting and I really love the time with the retrospection from today, a lot of work, a lot of challenges, which gave at the end a lot of positive effects. Going further to the overview, you can just see the number of deployment releases, because deployment was releasing one of the projects due to some security reasons, we were not aligned with the standard DevOps approach. But what is important in this chart that you can see how fast we were growing with the number of tasks. And it is also showing how we were expanding our competencies, team and projects within the first year of building this team in Makolab. So I was looking for multiple data that I will be able to show you. This is presenting the best the whole scope of growing, expanding and building the team based on the sample data from the deployment releases from one of the projects. Also, as you are able to see on the previous slides, there was a lot of work for the certifications. Just to confirm that we have the competencies and how the team was built. First I was providing an onboarding. It took three months for every single person. And after some time we discovered that the best person that will be providing the onboarding for the new joiner will be the person that was recently joining our team. Because that person has the best overview for the onboarding process. It was very effectively and also was able to integrate the new person to the whole team based on daily connections, not only with the team, but deep connection with the last onboarded person. So it was very smooth to join the team and also to join the tasks and competencies. After some time at Macolab there was a decision to run a bootcamp program. First it was for the Java, but after some time the decision was also to start it for the DevOps area, for the operations team. So we started that. It is also the three months program that we can hire juniors or even persons that were not familiar with DevOps activities before. And ensuring those three months we are teaching those persons in area of AWS, terraform Jenkins and it processes. And after that that person can join very smoothly the team and start working on the real production environment. As I already said, personality was more important than technical skills. And also for the team, a lot of trainings and certifications were provided. So how to integrate the remote team? This is when it's creating. I joined during the pandemic for building this team. The first was that all of the calls are with cameras on and there is no excuse for that. We work on a daily basis with each other. So we are not looking how someone is looking on the very specific day. We want just to see each other and see the emotions and be able to integrate more. Also after a few months I decided to have one on one outside the company area and discuss how the team is growing and what is the perception of the engineers about the team. And it also gives me a very big boost in the possibility to understand exactly what was going during that time. After that we are joining during the weekends, we are going to one of the cities in Poland and we are spending time together just to integrate more than only on the remote. If possible, I'm also available at the office. And from time to time someone is just joining the office to have at least few minutes of talk, even if I'm overwhelmed with the meetings, which I will also present later. So I building the team that feels the responsibility of the tasks of the common tasks, that those are not the tasks of the company, but tasks of the team. And everyone is responding for delivering the high quality. And also we have a trust for each other inside the team. What is not obvious, but also very important risk management. So from the very beginning also this is the culture of Makolab, of having the strong risk management processes. I was identifying and managing the risks, which I can say that 20% of the risks are connected with people, 50% were connected with quality and 30% with the customer or the competencies that at that time we didn't have in the team. So this process is quite critical for delivering the high quality and being able to scale up very quickly. From my management point of view, what I did to also have the team working together in the close knit manner. I provided some logo type for our team and also t shirts and some other gifts for the team. So for understanding that we are a team, we have something in common. And also I was promoting on the webinars internal webinars competencies and the team inside the company to let everyone know that there is something going on in this area and we are growing with the competencies multitasking. I will present on the next screen the sample week of my calendar. But all of the work for building the team brings me to the time that I have six or even sometimes seven meetings in parallel. So I was able to take two, one of laptop and one of mobile. There was an issue, I was able to listen to them both in parallel, but if someone asked me to speak something, it was quite difficult for having those two meetings. I work under it and right now I delegate some of the tasks to my team and also optimize the calendar on my side. From the internal development. Before I joined the team, I completed MBA studies and also it project management studies. And during the first both months I did personally quite a number of cloud and related certifications. And also after that the whole team was certified with the ITIL version four. From the external development I am talking for example to you about how the team was built up. And also in 2023 I was presenting a lot of talks on multiple conferences in Poland and across Europe. I built the process knowledge base and competencies inside the team. So we have the training program, the bootcamp program and also multiple documentations like the tasks list, emergency procedures, first aid kits, FTE reports and so on and so on. So generally based on the growing of the team, there was a need for having more and more data combined to having the information about how the team is growing, how the team is delivering and what we can offer for our existing and new customers. So also some acquisition, negotiations or transition of the processes, providing the information that hello, we are the operations team and we can deliver some added value for your projects. And it really gives us the boost in the number of projects for the recently 2021 and 2022. What is also important, based on the IT report from ITMT in Poland, they are dividing the leaders into four groups, people leader, business leader, technical leader and additional leaders, corporate architect. So I believe that what helps me most was that and I am more the people leader then I'm looking also on the business area and also due to the fact that there is no much time for that, I just delegate the leading, the technical part for the team. And I believe that they are specialists, there are engineers and they know what they do. I need to provide them the right environment for delivering the outcomes. Okay, I said that I will show you my sample calendar. So one or one and a half year ago it was looking like this. So there was no much time for me to work. I changed it, I moved some competencies to the team and also then I was able to take some responsibilities. So it was really demanding time at the very beginning to make it working, to build it, to make it efficient and to integrate the whole team together. How I can see the future. So right now we have one manager with assistance of line manager and we have a team of engineers. In the future, when the number of project and variety of competencies will grow, it will lead us to extract two competencies areas within one team and deliver separately the IT processes within one team and the DevOps processes within the second team. If there will be growth above, for example, 20 to 30 people, there will be also the third team. The operations which now in this plan is combine between the ITIL and DevOps teams on a different technical layer. What about the project? So we started with one customer and one project. We received the feedback that we are the first company that was taking care and was correcting 100% of the issues founded on production. Also under cooperation with developers, they start asking us for the help that they usually should receive from the corporate IT area. But they were so used to receive support from us. So even with those tasks that were coming to us and being honest with the customer is also very important. So if we find the bug, we inform the customer. If we find vulnerability on production that is not in our scope, we also inform the customer. And based on our competencies, we can advise other teams and say them what to improve. So this approach and this high quality leads us from one customer and one project starting 2021, to seven customers and twelve projects overall, eight running at the end on 2022. Also within the five tier, we provide the team and competencies give us the possibility to double the revenue from the first project. And within two years, the overall income for the operations team was tripled. I am not revealing the numbers from 2023, because it is not ended, but we can discuss it and hopefully next year I will be able also to show this area some statistics from one of the products that we are running. Those statistics are from the two years period. So weekly inside the applications we are taking care of 200,000 processes and during those two years we provide over 500 deployments. So it means that every single working day in average growth and deployment based on our productivity. Because we are also looking into the logs, the operations team, without any incident from production, was able to identify over front hundreds defects that the end customers end users were not able to see. And we were able to fix it before being hit by the critical incident of production or something like that. During that time we received over 1000 incidents and our SLA for delivering the solution was over 99.5%. And also as we were working in a quite complex environment with multiple other companies and applications connected to us, we did create over 600 incidents to other teams. And unfortunately the SLA was not as good as ours because it was around 95%. So usually we were struggling with the issues, sometimes that were not our fault and we were hited by other teams failures. On production. We also provided the independent monitoring of application on production, which brings us the possibility of having faster notification of possible errors, because from the customer side we're alarmed after ten or 15 minutes of having some issues. For example on production, based on our own monitoring of this application, we are able to identify the issue after 1 minute and start working to provide the resolution. Okay, that's all. Regarding the building of operations team, right now I will quickly switch to the sample projects that we are or were running for the customers. So one of these is service event assistant. I will not go deeply into what the applications was doing because from the DevOps, operations, maintenance and support, it is important to make the application operational and understand what the application is doing. But generally the business case is not always the most important part for having it maintenance on production. So for this project we provided the whole CI CD processes, including deployment of those processes, everything in 24/7 maintenancesupport and support on call model. It was based on AWS. We provided a lot of monitoring. For example Kibana Grafana Dynatrace and we are analyzing locks, including for example 1 million locks per week. Based on that, as I said, if we see any possibility to automate our work, we are providing automation with terraform and symbol cloud formation, but also scripting in bash or python if the tasks is possible to be done automatically. The next one from the aviation regarding the applications, which were the flight components with the whole ecosystem of for example communication between the ground and aircraft, Macaulay provided the whole development into scrum agile methodology, including quality assurance, UX and project management. We as can operation team. We are providing 24/7 maintenance and support in a call model and also expanding the monitoring, for example using data docker or using new relic with log analysis and also incident management based on the IT processes. The last sample that I have for you today is from the health industry. And here based on AWS and based on cloudformation, we did provide the full DevOps engagement, which brings us to the situation that we are building the environment from the very beginning. First, because it was important to deliver it quickly, so it was manually. Then we provided the cloud formation for building the whole ecosystem and expand for the new customers. And after that when there was some time for looking deeper into what was needed exactly for specific customers, we started cloud cost optimization process which can bring to lower the invoices with a really huge number, even over 50% after that time of optimization. Okay, going to the summary, I want to share with you some takeaways. The most important things that I believe drives us to the point that we are and being able to rapidly expand the operations team within the company and within new projects, new competencies and new team members. So the personality at the very beginning, in the long term is more important for the team success than technical level. If you need something to be done right now, today, and you don't have time for building the competencies, you should hire a senior person that is an expert in this area. But if you are building something and want to make it stable, it is in my perception better to give people the space for learning and take to the team the persons that will bring the same, maybe not personality, but the same level of understanding why we are here, what we want to do and where we want to be in the future. Small teams of course are more flexible and are being more able to be integrated and working closely together. And as I said, the role of line manager and the team leader can be like an intersection. And generally I really love working with the line manager right now and in the very beginning because it boosts up the possibility of expanding the team as now at the end, for the customer we can be one man army, but inside we are not one man but one team army. And we are working together to deliver the results for the customer, for the company and for the team. And we should understand that, that we are not an individuals that are making individual tasks, but those individual tasks are combining into some common added value that we can share with the customer and bring it and build and expand the cooperation with the customers. I always trying to give my team the broad context on the business, operational and other layers if possible, because the better we understand what is behind the task that we have to do, the better we are able to deliver it. So it improves quality and it is really working. So at the end, if you are building such a team, I will say that be more people leader than technical or business leader at the very beginning to build the common goal, to build the common sense of being together and working and delivering the tasks. And I think it will work not only in the DevOps operations area but also in multiple other teams, not necessarily in the IT industry. Okay, if you have any questions or you are in any stage of presented projects, you can contact me anytime. When everything around is on fire, it is sometimes good not only not to extinguish a fire, but to grab a coffee and have a short discussion because it can bring us to the good conclusion and we will be able to move the things and deliver something better than just the fire around us. So you can join me on LinkedIn and if needed I will give you assistance with your issues. Thank you very much. Jacobs Andra Makolab enjoy the other talks from 42 dev sec ops 2023 and see you in Ouch.