Hello, my name is Misuthrukal. I'm a quality assurance automation engineer and in this session I want to talk about the importance of quality assurance in DevOps principles. We know that DevOps is trying to bring development activities and operations together or close to each other as much as possible. And quality assurance is an important touchstone in this discipline. I'm starting with a beautiful breed scene, which is not a coincidence because quality assurance is trying to find ways or channels between different parties and make the delivery or the transportation possible between these parties or the stakeholders. So what we will talk about in detail is, as we just briefly started, we will talk about the importance of the role or the place of quality assurance in DevOps. And then we will start talking about the problems or the challenges that we want to cope with, or in other words, the achievements or the targets that we want to hit and aligned with these problems or challenges or the achievements or the targets. We will talk about several metrics or dimensions of quality and I want to focus at least two fundamental categories, which are the speed and the reliability or the health of our quality assurance activities. And eventually we will talk about monitoring activities and aligned with this monitoring, how well we can fit ourselves into the product or the project goals, and how well we can fit in the bigger picture, right? Because as quality assurance we want to contribute to the product or the project goals. So how well we can align in this wider product or the project goal set and how well we can fit in the bigger picture. Starting from the role or the importance of quality assurance, let's first of all consider what we want to do from a wider perspective. We try to provide the best solution to our customers, right? But what is the definition of best solution? So this is the role of Kuala assurance. We try to ensure we are developing a set of functionalities which will meet customer expectancies. So sometimes we try to communicate directly with the end users or maybe the product owners or the product managers to understand what the users or the customers need or what they are expecting from this product. And sometimes we are trying to communicate with the development teams to understand in what ways they implemented these features or the functionalities. So this is why the communication skills are very important for quality assurance members, because we communicate with lots of different parties or stakeholders and sometimes we try to bring all of them together on the same page. And thinking from the modern project management methodologies, we know that we are trying to reduce the delivery time and increase the frequency of deliveries or the deployments. Because speed is very important. We are competing in time and in modern project management ways we try to do the continuous deliveries or the continuous integration activities. So in this perspective, quality assurance is very important because whenever we have a neat release candidate, we have to ensure the quality, not only the accuracy of the functionalities, but also lots of different quality aspects. We have to ensure, for example, performance, usability, maintainability, several abilities of quality dimensions, right? We have to check so with a proper and health quality assurance process this is possible. Otherwise there would be some risks whenever we do not perform the quality assurance activities. Whenever we have a new release candidate, there will be some potential risks or vulnerabilities and this would be possible only with a good, or maybe with an automated quality assurance process set and aligned with this perspective. Of course, in a scenario where the quality assurance teams or members are not involved in the whole development processes, we can understand that there would be some risks because all the quality goals or dimensions would not be covered. And whenever the new candidate or the features or the implementation would be merged into the production environment, there would be some risks which would be resulting in some critical consequences. In the later stages, we will discuss why this is important to find the vulnerabilities or the potential risks as early as possible in the next slides. But first of all, let's try to understand what kind of problems or challenges we have, because doing the problem definition in the first place is important. In this way we can understand in what ways we can contribute or bring value by understanding the challenges that we want to cope with. So nowadays the products that we are testing are really complex and complicated, right? Because they are deployed in different platforms or execution environments, which means they have several integrations and interfaces as well, because they are communicating to different applications as well. There is a large network of applications and we want to do our verification and validation activities in this comprehensive scope with different constraints and limitations. Because there is the time constraints, there is limited budget, limited resources, there are some different constraints as well. So thinking about all these problems and constraints and limitations, we can derive some different quality metrics to understand how well we are proceeding with our achievements or targets. And of course they should be aligned with the bigger perspective, the product metrics. So we can start with think and Dora metrics, which formulates a good set of product metrics. First of all, we can think of the speed of delivery and the quality of delivery. We want to provide a good set of functionalities which will cover the customer expectancies, but we want to do that with a good speed, ideally the continuous integration, right? As fast as possible. On the other hand, there should be a good quality. We should talk about several quality aspects, not only the accuracy of the functionalities, but also that the performance, usability and several quality aspects. So our quality metrics can be categorized in two fundamental groups, the speed and the reliability, or the quality of our features or functionalities. So we will discuss these two categories in detail. And let's start with the first one, the speed of our delivery. So there are several aspects or parameters that will affect the speed of our deliveries. First of all, the direct parameter, which is the execution of test cases, right? Because it is directly affecting our pipelines or delivery time. If the execution of the test cases are slow, which means they are taking too much time. Of course, after some time, everyone will start complaining about the test cases. They will say that whenever we have a new release candidate, of course we want to execute our test cases. But if execution is taking too much time, they will say that they are slowing down the pipelines. So execution of test cases is directly affecting the delivery time. So what we can do is we can do a time analysis and try to understand the slowly running test cases, and we can try to understand the points where it's slowing down the execution and trying to get rid of the unnecessary weights or dummy weights in our test cases. Along with that, on top of the analysis, what we can do is we can parallelize our test executions. We can divide our whole set into some subsets, and we can make our test cases independent from each other. In this way, we can enable the parallelization of test cases, and it will already decrease the total duration of our test execution. On the other hand, even the reliability is affecting our test execution time. Maybe it's not a direct parameter, but eventually how flaky our test cases will be affecting our test execution time. Because if we have some flaky tests, which means one time passing and one times failing, then eventually it means we will need some retries to retry the failing test cases to understand if they are still failing or they will start passing. So this means the retries will increase the total execution duration and it will affect the speed of our delivery. So in terms of reliability, we can discuss different conditions. For example, if our features are properly implemented, they are implemented as desired. But if our test case are failing, then it means it's a false alarm, right? Because our test case is failing. But normally the feature is implemented as expected. So this is not a real bug, this is a false alarm. So in this case, we have to retry our test case, and we have to get rid of this flaky test case, the false alarm. On the other hand, if our test case is just passing whenever there is an unexpected situation in the feature, in the implementation of the feature, it means there is a silent horror in our test cases. There is a potential risk vulnerability. But our test case is just passing, they are not capable of finding the issue. This is a silent horror case. So which means we will maybe just let the issue go in production and we will have some extra cost in the later stages. These are the reliability issues and which we don't want to have in our test execution environments or the test frameworks. But why do we have some reliable issues? Why we would have even the false alarms or the silent horror cases in our test environments or test frameworks? Because we might have several test smells, as we may have several smells in our product code. We can have different test smells in our test automation framework as well. There are several kinds of test smells, like some of them are related to test coverage, or some of them are related to different maintainability aspects of our test code. But we can get rid of all these kind of test smells to reduce the reliable issues in our test automation framework and to reduce these test smells. Of course we need some strong quality gates. Whenever we have some new test implementation, we have to check all the code quality even in our test code. And if we have some issues, then we have to stop that, and we don't want them to let go in production because otherwise there will be some critical consequences. Maybe there will be some silent horror cases, or maybe there will be some false alarms stemming from this kind of testimony. So if we have some quality gates, which can be either static code analysis or doing the code review activities, the peer review activities. But by enabling these kind of quality gates, we can already find the vulnerabilities even in the test code and stop them before merging into our test automation frameworks. So the execution speed of test cases is already affecting the delivery, but also the reliability of the test cases. Because whenever we have a failing test, we have to stop the pipeline and try to understand if the failure is a real bug or a false alarm. And even the maintainability of our test cases is affecting the delivery speed. Because whenever we have a false alarm, which means whenever we have an issue in the test itself, we have to fix the issue in our test code. And until we fix and we make the test passing again, we have to stop the pipeline. And whenever we make the test pass again, then we can continue with the delivery. So this means how easily we can fix the issue in the test or how easily we can maintain our test is a parameter that really affects the delivery speed. So how we can improve the maintainability a very well known example is the UI automation, right? Whenever the layout of the page is changing, then of course the selectors or the locators of our elements on the web page should be updated in the test code. And if we follow good coding practices or some patterns that will contribute to maintainability, then we can easily update, for example by excluding the locators from the test classes or the spec files, then we can easily do that. By following these kind of good practices or different approaches, we can try to improve the maintainability because otherwise it will be slowing down our pipelines to fix the issues in our test code. One more good practice to improve the maintainability is removing the duplication or avoiding the duplication. Because whenever we have repeated core or duplicated code in test spec files, then whenever we need an update or fix, then we have to go to all relevant piece of code and we have to fix in all of them. But if we have only one source of truth, for example a dedicated class file where the implementation is done, then we can go to the relevant part and fix the issue in only one source of truth. These kind of different approaches or good coding practices will help us to improve the maintenance. So summing up the first part in terms of the speed, of course we can improve our execution speed by enabling or embracing the parallelization or improving the reliability of test. By removing the test mass or even improving the maintainability will contribute the delivery of the speed. Now let's investigate the second aspect of the quality assurance activities, which is effective testing. How capable or how good our test cases are to find the issues in the early stages because we might have test cases running very fast or we might have 100% coverage metrics, but still we might have some escape bugs or escaped issues which were not found in our testing activities but just reported from the production environments. But what might we be missing or overlooking? What might be the reasons for having these escaped bugs? Because normally we try to cover all the test cases or all the features with test cases, but still we might have these kind of issues. So we have to go back to verification and validation distinction. Sometimes we just talk to the development teams and try to understand what is being implemented, but this is not the only thing that we have to ensure. Right on top of that, we have to understand if the implementations were already aligned with the requirements in the first place. This is the distinction between verification and validation. What is being implemented and what was supposed to be implemented. So this is aligned with the customer perspective and customer focus. So we have to understand what was expected or required by the customers and what are we delivering or providing as a solution to them. So to try to improve this mindset, we can ensure that we are running the correct scope. We have to update our test cases whenever we have a new implementation or whenever even we have a new refactoring on the code base. Then we can try to check if our test cases are up to date, or if we are running any or if we are running any coverage issues. But not only the test steps or the test context, but also the execution environment is very important because sometimes, for example, let's suppose we have a data upload feature. So this might be running with different data types or data ranges. So not only running with certain data types like integer values or string values, but also running with double values is also important. Or not only with positive values, but also negative values are also important. So supporting the test steps with the relevant test data or running in the correct or supposed potential execution environments is very important because sometimes different users are running on different browsers, with different devices, or with different operating systems. So not only running the correct test steps on a certain environment, but also running on different potential execution environments is also very important. So one more thing that we might try to follow to improve our test coverage is trying to embrace different test design techniques. There are some different techniques like equivalence partitioning or boundary value analysis, not to overlook some corner cases or boundary values or different data ranges. To avoid these kind of issues, embracing different test design techniques can be a good approach or a good test design pattern. But also there is one more thing that we can adapt in our quality assurance activities, which is the personnel based testing. By thinking of different personas or different characteristics, we can come up with different scenarios, for example, thinking of different users from different regions or different ages, or with different characteristics, with different activities or motivations or frustrations, we can think of different use cases or different scenarios. So in this case we try to avoid having or overlooking some different scenarios. For example, aligned with this, we can come up with, for example, for different people, maybe the accessibility or the performance is more important than other aspects, or for other users maybe the usability or the user friendliness of the interfaces is more important. So in this way we can try to cover all the aspects of the quality, not only the functionality, but also all the non functional aspects would be covered by thinking from different perspectives and trying to understand all kinds of or types of different use flows in our product or on our feature. Lets we try to discuss the quality assurance improvements that we can do to contribute or bring value to the product perspective. Right? We try to provide or support the deliveries from two different fundamental aspects, which is the speed and quality, because we want to have a fast and high quality delivery and we try to support it from the quality assurance perspective. But how can we ensure that we are proceeding in the correct direction? Of course we can utilize monitoring activities and we can collect several metrics to understand how well we are proceeding in these dimensions or the aspects of quality. And we can talk about several different metrics. But one of the first metrics set from the quality perspective is the issues or the bugs that we have because it will directly give an idea about how quality how level of quality we have in our products or the feature sets. So we can show different distribution of bugs across their severity levels or across the component on which they were found or the module on which the bugs were found. But apart from those, I want to highlight the importance of having the post mortems which is kind of trying to understand or trying to interpret them, trying to talk to them, because bugs have several meaningful or valuable information. Understand. And if we try to understand what they are trying to tell us, then we can collect some insights and we can try to reveal some weaknesses in our processes and the product. And we can try to introduce some new initiatives and improvements for the future activities. Because quality assurance is not only finding the bugs, this is already good enough, but also trying to avoid them in the future activities. If we try to interpret the previously created bugs, then we can try to understand what kind of weaknesses we had in our past activities. And by trying to improve them, trying to create some new initiatives, we can try to avoid them in the first place in our future activities as well. So not only the bugs of course, we can call several other metrics as well. So what we can do is we can transform them into some visual graphs or charts and monitoring dashboards. Because sometimes interpreting the raw data is not easy. But if we have some visuals then we can easily understand what is going on in our environments and what kind of trends or charts we have. And we can try to collect some insights about the health of our processes and the product and the processes and try to improve them for the future activities as well. So to sum up, we tried to discuss the role or the importance of quality assurance in DevOps practices. We discussed the problems or the challenges we have and we discussed the ways to contribute to our product goals from quality assurance perspective. For example, we discussed how we can support fast delivery in terms of the execution speed. We try to improve the test execution duration or test execution time, and we try to reduce the number of false alarms because whenever we have them we have to stop the pipelines. Or whenever we stop the pipeline, we try to fix the issues in the test code. So we discuss the importance of improving the maintainability of test code because that would directly affect the time where we stop our pipelines. On the other hand, we discussed ways to support the quality of our deliveries whenever we have some reliability issues or coverage issues. We might have some escape bugs, but we discussed ways to embrace different test design techniques, or embracing different Persona based test techniques or different approaches to improve our coverage and the reliability of test framework or test environment or test set to avoid those kind of issues. And eventually, we discussed importance of continuous monitoring to continuously get some insights about potential improvements that we can embrace in our processes and the quality of our products. So thanks a lot for listening to me and if you have any questions or feedback, I would be more than glad to try to answer them.