Hi, I'm Hemang and I'm a software developer at amazon. com. I have a lot of experience in working with machine learning models, which are tasked to identify intellectual property. such as logos, copyrights, or trademarks. I worked on how to fine tune ML models, how to recompile them, how to deploy them in cloud environments at scale, you know, to the likes of 20, 000, transactions per second and setting up. Human assisted AI frameworks, to enable iterative development of ML models and also improvement in precision and recall of IP monitoring systems. So today I'm excited to give a talk on how human assisted AI is revolutionizing the intellectual property monitoring space. we'll talk about how human expertise, along with machine learning models, basically cutting edge AI, can help improve the precision, recall, and scalability of IP monitoring systems, thereby allowing, organizations to stay ahead of the curve and protect their intellectual property. Protecting intellectual property is one of the key challenges and one of the most important challenges for companies. Estimates show that globally companies take a one trillion dollar loss due to use of counterfeit goods and copyright infringement. what we've seen is that in the last few years, this number has been exponentially increasing. And that's majorly because of newer and innovative techniques or technologies that are available to bad actors. one of which is Gen AI. we've seen bad actors use generative AI to quickly, you know, generate content that's, infringing on intellectual property of organizations and put it out there on the web and, you know, other channels, other media, and, and thereby, you know, these, the losses have just been increasing. So because of these. Companies need advanced systems which actively monitor for, you know, illegal use of their IPs and, take action. This is where AI comes into the picture. There are numerous models available out there for different use cases that companies can use. in the IP detection space, let's go over some of them. One of the, one of the techniques is to perform embedding based searches. as opposed to keyword based searches, embedding based searches are far superior because, they're inherently performing semantic searches, whereas key keyword based searches are naive and basically just string searches, right? so. There are models, such as the CLIP model or, OpenAI's Dolly, which helps generate embeddings for images. These models can be used to convert any entity, you know, such as text, or an image into vector representation or something that we call embeddings. And these embeddings would have all the contextual understanding of the entity. Similarly, you generate embeddings of your query, your search query, and perform a similarity search. So this can be used for, you know, use cases such as identifying similar images and similar content to detect potential infringements of your copyrights. another method is using natural language processing. There are state of the art models such as BERT and Roberta, which are amazing in analyzing text. So these models can be used to, you know, if I were to give an example, identify if the usage of the term apple in a sentence. is in context to the fruit apple or the company apple. such use cases can help analyze text for copyright violations or trademark infringements. then we have the vision suite of models. there are models such as the YOLO family of models. YOLO stands for you only look once. so the YOLO V5 or YOLO V7 models and the newer models such as the VIT models, also, short for vision transformers. These models are really good in analyzing an image holistically. so these can analyze image and detect unauthorized use of logos, trademarks, and so on. Then comes the large, large language models. In the last two to three years. Large language models have picked up immensely. There's been a boom going on and these are basically models which have been trained on humongous amount of data, almost to the likes of the entire internet, right? so these models have, these models have enough context already. They can be used to solve really complex problems. You can just give it a research paper and ask it to do a plagiarism check. Or you could give them, give these models an image and ask the model to identify, you know, if the image has illegal use of a brand's logo and so on. But AI has its own limitations as well. AI cannot use, cannot be used in standalone because there's only so much it can do. Artificial intelligence is excellent at detecting known patterns and statistical anomalies, but it does not do really well in the unknown. if I were to give you an example, for example, let's take LLMs, LLM being trained on, you know, almost The entire internet's data. It would be really good in identifying, you know, the use of Nike's logo or adidas's logo, but if tomorrow I were to launch my own company and create a create my own logo LLM would not be able to identify the use of my company's logo in you know, images all over the internet because It would not have been trained on On on the data which includes my logo, right so ML models can only perform in cases where, they have context or they have, the, the, the knowledge on taking decisions. the other problem that, you know, LLMs is. The problem of hallucination, for the same query, for the same request, it can give you different answers in different times. Then, you know, that's when these are cases where, the LLMs or models are not confident enough to take a decision. So human expertise. A human is far, has far more superior contextual understanding and can take judgment in complex situations whereas the AI cannot. So what we propose is a amalgamation of human expertise along with AI's capabilities to create IP monitoring systems that are superior in terms of precision and recall. A very basic, approach to this would be, you know, a human machine collaboration where you let the machine take potential decisions, you let the machine take, make inferences, but you do not take, decisions. automated actions based on, you know, just the ML model. Let the machine learning model give you an output. And then a human reviews the output of this machine learning model. So when I say human, it would be an expert in that domain who is, flagging and, Re verifying the output of the machine learning model. So based on the human output, you can take actions, for the certain situation, for the situations. And also now you have a golden set of data that can be used to fine tune your ML model. And create the next version of your ML model. So this is what we call active learning where the machine or the model is actively learning from the human. There are two main advantages of active learning. One is improvement in precision and the other is improvement in recall. By improvement in precision, what I mean is the model is now better. at tasks which it was previously good at. So say there are situations which the model was able to identify, but there was some false positives, in the output of the model, which was audited by the human. The next version of the model will no longer have these false positives. So the model has now become better in situations which it was doing okay at before. This is what we call improvement in precision. Improvement in recall is basically the model is now able to identify new patterns, which it was previously not even able to detect. So these are better edge case handling that the model is able to do in the newer versions. If I were to give an example, say we have a logo detection model. the model was previously able to accurately identify logos of brand X, but not of brand Y. The human audits the data and annotates the images with, which have the logos of brand Y as well. This data is fed back into the model and the next version of the model is trained. The new version of the model will now be able to identify logos of both brand X and brand Y. So these are newer situations that the model is now able to identify. So this is what we call improvement in recall. what we generally see is when we compare, you know, just a ML model, in contrast to a framework where we have a human AI, human AI setup. in the human AI setup, we generally see that there's a 20 percent increase in precision and almost a 25 percent improvement in recall over time. And these numbers just keep increasing. Increasing with more the number of iterations there are. So how do we make this human AI framework more scalable? There are two components to it, right? the human component and the AI component. To make the AI component scalable, the best strategy would be to deploy your machine learning workloads on the cloud. There are multiple cloud service providers out there, such as Microsoft Azure, Amazon Web Services, Google Cloud Platform, and these have numerous services, which are at your disposal, available out of the box for, different parts of this pipeline. for example, they offer compute services where you can containerize your machine learning workloads. Or, ML code and deploy it on instances which can be auto scaled based on your traffic. At times of peak, the number of ML instances increase, and then there's no peak or there's hardly any traffic. Your ML instances threat use, this is not really possible in on premise solutions. they offer a lot of data integration services as well. So you could host all your human audit data on, NoSQL databases in the cloud and this, these databases can automatically be ingested or linked to your machine learning, instances or machine, machine learning pipelines on the cloud, which act, and this data could act as the training data for every, every new version of the model. Now, how can we make the system scalable? from the human angle. The goal over here is to reduce dependency on the human as much as possible. And the human should have the least number of human touch points as possible. so what happens is that over time with few, few iterations of the system and few model versions, you would see that the model is performing really well with high precision and high recall. in certain use cases. for example, if I had a logo detection model, the My model would start performing really well in identifying logos on shoes But it might not be very well on identifying logos on bags so now A human no longer needs to see the output of the model For where all the images are that of shoes because you have a high accuracy in that Product category, and you're going to let the model take automatic decisions and actions for those categories. So what you then do is you let the AI flag what is important for a human to review. So in the shoes category, you would let AI to take automated actions. But in the backs category, you would park all the decisions for human audit. You would, accumulate all the human audits across all the different product categories, which the model is not able to accurately determine. And then you would have an algorithm that prioritizes the human audit. You would pick, you know, categories that are important for your use case, and then let the human only review those use cases or those, audits. Okay. And so now you have technically reduced the number of audits that a human has to do, and you feed that into the next version of the model training. So thereby you have now reduced the dependency on the human as well and made the system more scalable. by doing this, by offloading, you know, What we generally see is by offloading 40 percent of the workload from the human, you're still able to maintain 95 percent of precision in most cases. So, yeah, in a nutshell, the human assisted AI, combination. Has a lot of benefits, especially for brand protection and IP monitoring. we see improved precision, high amount of, recall and very less false positive rates than, you know, a human assists an AI. So what should be the mental model of someone who is implementing a human assisted AI framework for IP detection use cases. So step one is to. Assess the entire IP, IP evaluation framework as it stands today. Identify which parts of your system are good at taking automated decisions, and which parts are not that good and need some input from the human. because only the parts which have high false positive rates, would be the parts where, Step 3 would be to develop an integration strategy, which basically means that, like, where would a human audit the data? You would need some sort of a user interface which outputs, which displays the output of the model and where, where human auditors can come and audit the data. All you then need to figure out what the output data of this audit would look like. And it should be seamlessly integrated with the ML model. What the, the form of input that the training of the ML model can take so that all of this output is readily fed into the system, for training of the next version of the model. You would then have to train the teams on how to audit this data and finally train your ML models. The audit and, retraining of the model should be monitored and repeatedly iterated over to see like which parts of the system you have achieved the accuracy that you want so that human test points can now be removed from those parts of the pipeline. So yeah, that was the end of my talk. thank you so much for attending my talk and feel free to, reach me on LinkedIn if you have any further questions about this. Thank you.