Hello everyone. My name is Vipin Jain and I am speaking from a beautiful city of India called as Jaipur. It is also called as the pink city of India. As already told in my introduction, I work for a company called as Metacube Software. As a senior delivery manager and head of the QA department. IoT is my second love after my job as a delivery manager. And this has been always a challenge in front of me. How to develop a strategy to test IoT end to end. Now, end to end testing, we all know where we do the system testing, the integration and unit testing. But since IoT is not just a single software or a single hardware, it is a mesh of hardware, software, lots of protocols, there are APIs, there is a huge bandwidth connection between them. So I always ponder about how to test or how to develop a strategy for embedded systems, particularly Iot. About end to end testing. So in my talk end to end testing strategy for embedded systems, I will give you certain suggestions from my experiences. So let's move forward now. First and foremost question that comes to our mind is why do we even need to end to end test IoT setups when we can test each aspect of the application? So we can test software separately, we can test hardware, protocols, firmware, everything can be tested in a separate way. So why we need to perform or develop a strategy to test the IoT application end to end? Because we do not want to be in such a situation. This is a brilliant example of evolving an end to end solutions without developing a strategy. And you can end up in such a situation. So let's begin by a very brief history. So in the very early phase, IoT was developed as a two layer architecture where you have lots of devices and then you have lots of sensors or the things. IoT is Internet of things, so lot of things which produced data, and there is a two way combination between them and the cloud and the devices. As the application started getting complex, the IoT infrastructure or the architecture also started becoming complex. And then we introduced the third layer. And hence IoT became a three layer architecture. Where there is an application layer, a transportation layer and an interception layer, the things become more complex, the infrastructure becomes more complex, and similarly the software has become more complex. And ultimately, after lots of, I would say development and lots of thinking, a new seven layered architecture was introduced. And this is the one which is currently being used. If I quickly take you to what this seven layers architecture means, these are the seven layers where the lowest level or the things which produce the data, they are the physical device layers. Then there is a connectivity layers, edge computing layer, data accumulation, data abstraction, application layer and finally the collaboration and process layer. Now all these layers were introduced because the modern day applications are very complex and lot of communication happened between layers. So then in order to develop it or in order to test it, we need to apply our logics at every layer. And this is the base of my strategy. Now, what are the challenges in IoT testing? As I've just explained, IoT is a multilayered system. Then there are lots of dissimilar technologies being used in developing these systems, changing from very low level microcontrollers and very high level server programming. The functionalities span across multiple layers. Hence the communication between two layers become very important. Then the devices follow protocols, but then different protocols can be applicable to different devices, which means the protocols can become incompatible. The functionalities were developed by different teams or the teams which are sitting geographically very apart. So hence, because those teams are not working together, sitting together, there are scenarios where the use cases, they do not match with each other. The solution looks simpler because there is a very limited UI, but below that interface there is a very, very complex architecture and code is very less to justify your testing because I can write 100 test cases and can say that look, everything is working fine, but those 100 test cases are actually just testing the code which is written, which can be only 10% of the entire architecture and not enough test cases are written to test that architecture. So these are some challenges which are different from the challenges we have faced in testing noniot applications. So what is the general practice when we start talking about e two e testing? Organizations use e two e testing at system level. They follow the most logical option available, like assemble the system fully and then test it, which gives a realistic simulation of the end user experience. But the issues that they face, as I explained earlier, also systems cannot simulate all the situations. Building such a system is very time consuming and costly. You need to gather every component, make sure all the components are connected with each other, the protocols are compatible and so on. Testing become very changing as it involves many applications working in tandem. APIs become very, very important here. The error detection happens in late cycle because everything is made, then it has been simulated and now if you can find an error then the entire setup or entire development process Iot needs to be reworked upon. This ultimately results in a lot of delay in the time to market and the responses to that application are very verified and they become very difficult to handle. I take a very simple example which is called as a blood glucose tracking system. Now what happens here is, as you can see in this diagram, there are two variables. One is the glucose sensor and the other is the insulin injector. And then there is a smartphone which acts as the middleware. So very simply put, there is a glucose sensor, like a small, I would say a patch which has a chip and that is inserted on. There's a small pin inserted inside your stomach and the patch is stuck on your stomach. And then it is constantly checking the blood sugar level. If the sugar level goes above or below a certain threshold value, it sends a message to the mobile. And from the mobile which acts as a middleware, it goes to cloud where lot of healthcare management system takes place. And as a result it sends an instruction back to the insulin pump if required, so that the pump gets activated and some insulin is secreted inside the body of the patient. So he gets immediate attention despite of where he is standing or sitting in the world. So this is the blood dugout tracking system that I want to explain. So this sensor monitor the blood for certain parameter, and the insulin injector follows a programmed schedule. But the problem is, if it is disconnected from the other components of the system, then there can be issues, right? And we'll see each issue one by one. Let's see this diagram in detail. So now you can see all the movements of the data and the instructions between various places. And let me explain each one of them. So, as I mentioned, the smartphone app serves as a middleware. And in addition to providing simple analysis to the user, the app forwards glucose level information to cloudbased healthcare system for additional processing. This cloud application compares the current measurement to the historical data of the patient and perform advanced analysis. And IoT constantly keeps looking for some unwanted patterns. If any unwanted pattern is found and IoT can lead to a potential danger, the system sends a warning directly to the user or reports to a human medical staff. Specialized medical consultant can immediately analyze the data and decides whether or not a special alert needs to be generated for the patient. The patient receives an alarm and guidance for the next steps, which can include change in the medication dosing. After the patient's approval, the change are delivered to the injector. But there can be a scenario where the patient is IoT able to respond. In that case, the medical consultants can send the GPS information packaged with the smartphone data to send an ambulance. Your mobile's automation can be tracked and immediate medical help can be sent. Right? So this is the entire blood tracking system diagram which I just explained. So this healthcare system covers everything here the phone acts as the middleware and immediate helps from the doctor to the patient. Can happen if the healthcare system alerts this and all this data gets recorded in the cloud, so that the next time something happens, it is again matched to the historical data which is already present. Let's see some wider scenarios. So what is the basic use case here? The blood sensor is simulated, the data package is selected, the data moves to the cloud, the cloud generate alerts. After comparing the current data with the existing data, the medical staff responds to the alert either by injecting the insulin or by changing the dose of the insulin, or by alerting some medical staff and ambulance nearby. The patient also receives those alert and notifications and ultimately the injection schedule and the injecting of insulin is completely automated. And all of these things still have to follow, still have to cater to these market requirements. Test early and test often speed versus quality time to market automation. We need to still perform all of these. But this is a typical IoT application. And if you see there is a cloud storage, there are data moving, there are bandwidth involved, there are devices, there is mobile, there is Internet speeds, all of these need to be tested. And then only we can say that I have what is called as an end to end testing strategy for testing blood glucose patient monitoring systems. So what is the solution, how to perform these kind of testing? Deconstruct the system into layers for more effective testing. So rather than taking the entire application as a whole, break it down into perception layer, network layer, middlewares, business layer and application layer. Your perception layer consists of the physical objects, sensors. The network layer consists of the transmissions, three g, four g. The middleware layers consist of information, process, action, the storage, whereas the business layer, it takes a look at the analytics and flowcharts to see the data. And finally, the application layer where the smart applications and their management resides. And then in each layer, isolate the components and test early. So if every layer just separate out the components. So your wireless, IoT devices, IoT smartphone, laptops, security gateways, security devices, just separate them and then test each one of them. Few are already required as tested by their vendors and few we have to test as testers. Right? So this is the two things that I want to explain. And now let's move forward. Deconstruct the system into layers, which is the first one. What are the primary challenges? Designing the system in such a way that it can be conducive to deconstruct in smaller blocks with well defined interfaces. Just like we say that in manual and automation testing automation needs to be built into the code so that when we automation engineers want to perform it, then they can do it. Similarly, the system should be designed here in such a way that it can be broken down or deconstructed into smaller blocks, and each block should have a well defined interface. And then you have to build automation around these blocks. And there is always a discussion that goes on between unit testing versus functional testing. So in general, a test plan should include a combination of unit testing, integration testing, and endtoend testing. The proportion of unit test to integration test may vary depending on the complexity of the solution. The more complex the solutions is, the more important unit tests become, because as the complexity grows, it become more and more difficult to simulate high level interfaces that ensure that various paths are executed. Unit testing, though, is expensive. We all know that, and it is expensive in terms of time and resources. Therefore, someone with a programmatic experience, for example, an engineer, must write the test, and because they are closely tied to the code, unit tests are also brittle, which means changing in a code could easily impact them. So an engineer is required consistently to maintain the tests as you move further up the stack. Function level tests are less prone to breakage, but it becomes hard to identify systematic issues when a unit test fails. On the other hand, identifying the root cause is very simpler, so you have to actually do a trade off to find a balance or create a blended approach to see a balance between unit test and functional testing. So use a blended approach for testing IoT, deconstruct the system into layers, right. So what we want to discuss here. So for IoT solutions, the first natural layer contains the wireless communication components. This is where the subsystem interacts with the APIs. Now, underneath the APIs are messaging protocols such as MQTT or HTTP, which send payloads like JSON and Rest, as well as the proprietary protocol and binary payloads. In the IoT world, the communication usually follows a publish subscribe model or a request response model. This model involves broadcasting data while other components listen for and then consume the published data and perform an action in request response model. We just send a message to the server directly and asynchronously waiting for a response. Automation is the key. What does that mean? To do this, you have to have a framework for simulating the sut system under test and verify its response, as shown here. For simple scenarios, Python or simpler steps can serve as service testing solutions. There are a number of stepping utilities that easily allow you to send payloads for testing purposes, but this tactic doesn't scale as the number of test scenarios grows from few hundred to few thousand, adding and changing the scripts become difficult and inefficient. So implementing more complex parameters is very difficult to do. With simple scripting tools. For smaller small sized things, it becomes very useful. You also can automate the server components. For example, you see this cloud healthcare system. There is this API layer. The message or the payload is sent to the services. The response comes back and then from the cloud two messages can go, one to the medical point location system and other to the medical staff consultation center. So this is a back end, right? The database contains all the historical data. The process is similar to testing the sensor. Simulate the system by sending a simulated package of data from the patient. Now the blood scan report, which comes back if it has some deviation, which means a warning needs to be produced. High priority warning involves human medical staff, low consultation. The warning message is sent back to the patient may contain information about the nearest hospital, urgent care or other medical facility. The testing framework will execute a number of test cases containing different values for different patients and expect specific warnings to be generated in the response. We can isolate the components and stress early, so let's also use a case here. Scan blood to determine glucose level we could potentially place the sensor probe in a liquid with a known level of glucose and other parameters corresponding to real blood, but this approach is neither practical nor it is scalable. A better approach would be to intercept the function call that reads from the probe and redirect it to the testware stub or mock for generating and simulating a response. It will eliminate the probe itself from the testing process while enabling the opportunity to perform automated testing of all the other parts of the system. A mock of the interaction needs to be installed to emulate a hardware function call. The stuff should be able to respond with reasonable values during the test process, using hard coded values or reading test data from an external data source. You also have to isolate the server side components, which again looks like the client side components, but there is some difference. For simple test cases, you can use a stuff that contains a node JS script to produce the standard responses. But for more advanced scenarios, a service virtualization tool can allow you to define a response depending on the input pattern. You can also use service virtualization to record actual traffic and replay it to simulate the real operations of a connected system. In some cases, you may want to switch even between a real system or a virtual asset. So a group of medical experts here are connected to this prioritized queue for consultation requests. The first person that becomes available, processes the next highest priority request to enable the automated testing, we need to replace this part of the system with a virtual equivalent. So this is how we can separate the components and then test them separately, whether they are the server side components or whether they are the client side components. So you have to remember a few things. IoT systems require thinking about the software quality in a larger scope. IoT solutions, such as our medical device examples, they are different from the normal systems because an individual feature may span multiple layers of the solution. Which means if we start testing these individual functionality, you actually have to traverse between various layers. That means the communication between any two layers becomes very important. The response request models or the published consume models are again very important because data gets published from one layer, may get consumed at some other layer. Similarly, the response sent or the request sent from one layer may receive a response from another layer. So at every layer we need to ensure that we should be able to send data or we should be able to receive data. Then delivering a high quality system, it requires testing capabilities at every layer. But the problem here is the low level layer has a lot of c code, the API testing layers, and then there is a backend part which is the very hard to access layer. So you have to develop strategy to cover all of these. You also have to consider a lot of cost which is associated with the system, because the reason is quite simple, a design failure that can outweigh the cost of deploying or testing solution. If you encounter such, ensure that you have a testing solution which enables you to isolate the test components or the API testing or the backend testing. So with this I come to the end of this talk. I hope you got some valuable information out of it. IoT testing is challenging, but it is not difficult. You have to upgrade yourself from your normal testings to IoT testing, and you also have to upskill yourself from testing softwares to testing hardwares. Firmwares have a mix of all kind of testing, and then you should be able to test Iot quite easily. Thank you. Thank you for hearing me patiently. I'm open to any kind of questions. Thank you.