Hello. Welcome to AI with JavaScript. There's a great chance that you've come to this talk because you want to learn how to develop your own AI applications, all with JavaScript. And there's also a possibility that you want to be able to use JavaScript because maybe you don't have much of an experience with data science or machine learning. And I want to say to you, that's awesome. Because. Although you may not have a, like any experience with, with machine learning, or if you're not very experienced with machine learning, that doesn't mean that you can't actually develop applications that can work with ML models. In fact, I am definitely not an expert. I'm probably just like you, I work with people who I consider to be machine learning experts, but I am definitely not one of them. However, the great thing is that so many libraries exist now, they make it so easy for us to get started, to get to learn, to build our own applications and from there, can like Catapult us to learning more, right? Can let us go deeper into learning how to work with machine learning, how to create our own models, how to train or how to retrain existing models. And so two of the models that, sorry, skipped over to the code a little bit sooner than I was supposed to, but two of the models that, two of the libraries that, can actually help us with this in JavaScript in particular are TensorFlow. js, which was created by Google and Transformers. js, which was created by HuggingFace. Great thing about the both of these libraries is that they're open source. They allow for machine learning on the client side. So we don't actually have to work with the server to actually be able to create our own applications with these models. Again, the models have been pre trained. They have Been trained on an existing data set. You don't actually have to provide your own data sets in order to be able to work with these models. So depending on the type of model that you're working with, and depending on what you need to provide, you wouldn't necessarily have to provide. you wouldn't have necessarily have to create a data set or clean a data set or anything like that. also TensorFlow and transformers are two. Like libraries in Python, so you have TensorFlow and Transformers, the Python libraries, and they have TensorFlow JS and Transformers JS, the JavaScript libraries. So what that means is that if you see a model that has been created, like with the Python libraries, but you want to try to use them in JavaScript, it is possible that they could be converted for use in Python. In JavaScript really depends on like the type of, like the type of model that you want to work with and if that is supported in a JavaScript library, however, it is possible in both libraries actually provide like a conversion script that you can use. We can work with these libraries like we would any other like JavaScript library by installing them via a CDN npm package and both actually do allow for server side inferencing with node. js as well. So just to talk a little bit more about like why machine learning in the browser could be a good thing. because, these libraries, they take care of a lot of the work for us, but it is a good idea to know exactly, what type of benefits they potentially provide. And One is that they can increase privacy for our users because we wouldn't have to, Like we wouldn't have to send their data elsewhere for, use with the models that have been developed, their data will stay So, these are all things that you can do. But the last thing I want to talk about is that you can do a model that is running on their device. And this is something that I think a lot of people are very much concerned about. And so it can be really important, especially depending on the type of application that you're working with, and the type of data that you are, you're working with and how like you want to keep it secure. working with a model that is run on a user's device in the browser could definitely Be useful another thing. that's pretty great about working with models in a browser is that it can reduce latency because again, you're not like, you don't have to send, the user's data, their information elsewhere, and then get a response to them, be able to provide them, with. Whatever that response may be, right? If you're working with images, maybe you provide like information about the type of image or maybe you provide text about the image. depending on whatever the response is, you don't have to worry about it. Then like being returned from an API, for example. Which kind of gets to another great benefit of working with machine learning models in a browser is that, the models can still be run without an internet connection. So if you have, an application that maybe needs to be a progressive web app, this could be, a really good use for working with machine learning models in a browser versus, another, alternative, like working with it on the server. Another thing to note is that deployment could be much easier if you are working with a machine learning model in this way, because you would simply, deploy the application like a static site, like any traditional static site, you wouldn't need to develop an API or anything like that. Could just, run as you normally would. However, With anything, there's always going to be some tradeoffs and drawbacks, definitely, I want to keep these things in mind when thinking about, whether or not working with, ML models in a browser is the best thing, to do depending on exactly the type of application that you're trying to create. So something to note is that once, a user actually, once like a model is initially downloaded onto a user's device, it could be a little slow the very first time because, to have to download it, it has to, the initial load can take some time. that could take a few minutes. and especially depending on the type of device that they're using and the network connection. That they have or where they are, that, that initial load could be a little slow, but, after, some use it, that, that should go away, models, when they are running the browser, they could be extracted or even potentially copied. great thing is that in this particular case, we're working with open source. libraries have open source models. this is not a concern with these two libraries. However, depending on, what type of models you're working with, this is something to keep in mind. Maybe if you're working with custom models, and you're looking to be able to work with them in the browser, that is something that you may want to think about. if a model is pretty large, then it probably wouldn't be best to use for, I could probably be best to use in the browser in this way. just because you could run into some performance issues, especially depending on the user's device. And, something else to note is that when working with the machine learning models in this way on the client side, you have to think about, like, how are you going to update them because, they are definitely going to have to, be updated. You don't want them to go stale. And then you also want to be able to keep in mind, exactly how will you handle version control for those models. So let's get started talking about TensorFlow JS. Three things that you can do with TensorFlow. js is, you can work with an existing model. as I mentioned, those pre trained models, they've already been trained on an existing data set. You don't have to actually, provide data in order to be able to work with these. however, TensorFlow. js. There's also the possibility of using transfer learning to retrain models into Cisaflow JS with your own data. So if you see a model and it, it has like a type of a task that you're interested in, does something that you're particularly interested in, you could potentially retrain it with your own data so that you wouldn't actually have to create the model from scratch. Thanks. But just actually just change the data set. however, it is possible to create and train your own models with TensorFlow. js as well. as I mentioned, you're, as a beginner, you're probably looking to work with existing models, get started. but then as you're becoming more, as you become more familiar, And more comfortable with working with, machine learning models in a browser, or as you just want to dive deeper into how to, get more familiar with the different types of models out there and how to, train them, then you could go to retraining a model, as your next step. And then, from there, you decide, Hey, I am ready to go. develop my very own model. And so then as that last like step there that you can definitely do all with this one particular library. Some things that you can do in TensorFlow. js are like image classification, object detection, post detection, and even speech command recognition. So I'm on the TensorFlow. js website here at the moment, and I'm here on the page for the models page where you can see the different types of models that are available for use on, for use with this library. You see how like different types of vision models, as I mentioned, image classification. Object detection, we have models that can work in particular with the body. So you have like face detection type models, pose detection, hand pose detection. There are different, like text models as well. So you can, have models that work with question answering. text, they have a text toxicity detector, so you can check to see, if the content that you see on social media, for example, the text that you see on social media, you could actually use this text toxicity detection model to determine, is it toxic or is it not? There's also audio. So here we have the speech command recognition and that particular model, that you can work with. So there's also, you can go to, the trends, not the transfer flow hub. Sorry. You can go to Kaggle. com models to see the TensorFlow JS model hub. so I'm on the website here and I'm going to select framework and I'm going to look for TensorFlow JS. So these are all the different models that actually can be used with TensorFlow JS. and there, there are several more. I'm going to select this first one here. It's an object detection model. And as you can see here, it provides more information about, the model and what it does. there's more details. click read more. You can see how this object detector, how it works. it shows car, table. and if you continue to scroll down, you'll even see like how to work with it. There's some code examples. So if I come down to TensorFlow. js, here we are, and then provide more information about like how you can work with it here. So now what we're going to do is actually look at a demo, that I've created for, this particular, that I've created with TensorFlow. js for this, for this talk. And as you can see here, we're going to start off with some very just simple HTML, very much what we're all pretty much used to, here, I'm just going to have, I'm going to upload an image and then I have a container where I'm actually going to, place that particular image that has been uploaded. Here you can see we have the script tags that are installing TensorFlow. js and then also installing the particular image classification, image, image classification, model that I'm using. Sorry. Thinking image classification because I have worked with another one before, but actually this one is supposed to be the object detection model. My apologies. So here we have the TensorFlow. js object detection model. And then here I'm just importing this app. js. I'm just calling this app. js script here. Here we just have plain old vanilla JavaScript. What we're going to do is we're just going to listen for the image once it has been uploaded. That's it. And then we're going to create a URL from that image. And then we're going to, Add the image to, a container, like the image container that I showed, in the HTML. And the reason why we're going to do that, because with this particular model, it will not work if you send the, it didn't, work when I tried to send a file directly over, it didn't really enjoy that. we're going to create, an image element, and then it's going to give it an ID if I upload the image so that we can easily, grab it. And then I have this, this function that I've created called getImagePredictions. And so this is where we're actually going to have the, work with TensorFlow. js. And as you can see here, we are calling the particular model, that was loaded. So if I go back to the script tag, you see that name, CocoaSSD. So here, CocoaSSD, we're going to first load the model. And then, from there, we're going to use this detect method and we're going to send the image that has been uploaded and it's going to make a prediction. And so I want to display information about that prediction. I want to display the prediction information in the, in the browser, so that the users can actually see, the information about the image. And so what we're going to do is, we're going to take the class. That's returned from the model. So the class will be like the name of what is seen. So for example, ball would be class in this particular model. So that would be part of the response there. and then there is a score. So the score, will, Indicate like how confident the model is in its actual prediction. And then we're just going to convert that to a percentage. So it's a little bit easier to, see and understand, when we look at the demo and then here we just have some styling that we're gonna do to be able to display the predictions. and in here, we're going to use information from that response. in what this particular model is called, like it's labeled as B box. So there's, it can be an array, that has information about each, like multiple objects in an image. And inside of that, you'll have like the B box. And then we'll provide, the position, select the left, the top, leave also the height and the width. And so we're going to use it to help us better position, the predictions inside of the image. So now we're going to actually, upload the image into, in this app just to show how it works. Upload. Going to upload this image of Messi, the soccer player. he's holding a trophy. I'm imagining this is probably after the World Cup win. and so you can see here, he's like holding a trophy and that's him. and so the model returned the class of person. So it has detected a person inside of this image and it has like a 60, 66 percent confidence. in this prediction, probably need to improve to see it. I need to improve to see it. It says a little bit more so that the label, so we naturally see a little bit more on messy versus the trophy here. so it looks like this particular datasets mainly focused on people, doesn't recognize, a trophy, however, what can be done then when. Would be to retrain this particular model with a data set that has more images, like images with trophies, or you can even have one that maybe have a data set that has like images of various soccer players, and you could take this model and train it so that it could detect various soccer players. So instead of, it returning this person, it could return, like messy, for example, or soccer, if you have. A data set with players. If you have one, it's just maybe with athletes. Maybe it could determine that, as a soccer player. So you could definitely be able to find a way, to provide like more details by, training, retraining the particular model with, a data set with even more, images to learn from. Okay, so next we're going to now talk about Transformers JS and learn about how we can work with this particular library as well. It also works with pre trained models. So he, these models have already been, already have been trained on data set, which is really great. because again, as a beginner, you can just choose the particular type of model you want to work with. another thing that is really cool about transformers. js is that it can be used with custom models that have been converted. So I mentioned earlier that, they're like Python, like there's a transformers and it's TensorFlow, Python libraries, and those models can be converted. But also my understanding is that with transformers JS, like if you're working with a team, and let's say maybe someone actually creates their own particular model, maybe not even with transformers, or, PyTorch is usually pretty common with transformers. But my understanding is that there is a way in which you can convert the models on these custom models that can be used with transformers JS. In that particular library, like in the browser as well, some of the tasks are supported by transformers. J. S. or natural language processing or an L. P. such, like your test classification, your translation, you have computer vision. With image classification and object detection, automatic speech recognition and audio classification. And we even also have like multimodal tasks that will support it. So an example of that would be zero shot image classification. So with image classification, a model will be trained on a particular data set. and so then it will recognize different types of different types of, what we just saw with the TensorFlow JS that demo, like how we recognize the person. However, with XeroShot, these types of models could then, also be able to recognize images that they maybe haven't been trained on, the dataset doesn't include, doesn't include. A particular image in a data set, but it would still be able to recognize. So like with that image of Messi, where he's got the trophy, maybe if we were working with the multimodal model, maybe it would have recognized the trophy. Another thing, to note about Transformers. js, that I think that makes it really, beginner friendly. And to be honest, this is just my opinion. I think it may be even more, a little more beginner friendly than TensorFlow. js. just a little bit more beginner friendly. But I still think both are really great libraries to work with. but something that I think that makes Transform. js pretty easy to work with is that it has this pipeline API. And so what that does is that it groups a lot of the, it groups like the inputs and the outputs, the pre processing of the inputs and the post processing the outputs together. So as you can see, like in this example, there's a lot less code written here in comparison to that demo, that I had with TensorFlow. js. So here, you're just going to, import this, the pipeline from the library. and then here with the pipeline, you can see I have let pipe equal await pipeline and then sentiment analysis. So here you're actually going to declare the particular task that the model will, will run, like what the model will do. It is possible. To specify a specific model as well. However, if you don't do that, then HuggingFace, the TransformerJS library, it will just choose a default model for you. So here we're doing a sentiment analysis task. And so here I'm just going to enter some text, in this particular example. So COM 42 JavaScript is a great conference. And then it could return And, returned the, a score and a label here. And it would be a positive, in this particular case. Yes, it is a positive. it could also like return like negative, for example. And then it will return a score like, how positive or, negative the particular test is, when using this particular sentiment analysis model. Like TensorFlow. js, HuggingFace also has a hub where you can actually choose models to work with. And here we are on the website here. And so I'm on huggingface. co slash models. So the models page and you can then filter by the type of library that you're working with. So I'm going to select transformers. js here. So we can only get models that will only work that we know for sure will work with transformers. js. I'm going to go back to task and so I'm going to select. let's see, let's say text to video hashing. There aren't any texts to video, so let's try image to text. So here we can see that there are, 14 models that have, that are currently available to work with. The transformers JS library that, would, take an image so you could take an image and then the model, but then provide a text description of that, image. So if I click on, let's see, this particular model seems to be pretty. popular. And here you have like more information. You can see how, many downloads is received in the past month. You can see more information. this one doesn't have a whole ton of information about like exactly what this model would do, but if you click, use this model, this click on transformers JS, it will show like an example of how you can actually, work with this particular. Model. So here you see with the pipeline, we're declaring the task of image to text, and then we are specifying the model name, and then from there, we can go on and work with this particular model. Something I have noticed is that hugging face will often provide a link to maybe like the transformers version of the model. If there's not a whole lot of information, if you need like more details. However, there are some that can have more details. I think I saw like an image text to text one that provided Yeah, this one's a really popular one. So it has like more details again It also has the use this model so you can still see how to we're up in this particular model with the pipeline API. but then it also has some more code examples here in the detail as well. and it even has an example kind of showing how this looks, when working with this particular model, which is really neat. So I have a, demo, so I've created a demo for, working with transformers. js as well. with this particular demo, we are actually going to upload an image. I'm going to upload an image and then we're actually going to, use the text to, the image to text model, to be able to provide a text description of the image that has been uploaded. Okay. So a little different from what we did with the previous demo. a thing I really like about HuggingFace is that it has something known as spaces. And so with a space, you can actually upload your code and then it will create the, then it will create app for you is deployed automatically. so you don't have to actually do anything more than just upload your code. if you're working with. If you're working with, GitHub, if you use GitHub to, store your code remotely, then you can actually, connect your GitHub repository to a space so that you can actually just upload it in one place. in this particular, example, I ended up just I'm going to be sharing it, uploading the code directly to the space. And so you can see it was very similar to any kind of like remote repository of like your branch. have the different files here. And so start off with my HTML. it's very similar to what we saw with the other demo here was have a place where we're going to upload the image, have um, container to show the image, maybe display, an error message if there's an error. And then also like the, we want to, display the description of the image. As you can see here, I am not actually, I'm not, I don't have a script. For the, installation of the transformers JS library here, actually, when we get to the JavaScript file, you will see that it is actually being called. So here we're using, I'm using the CDN here again, just, this would be a little easier to make it a little bit quicker for the demo. but you can see here, we're call, we're importing the pipeline from the, via the CDN here in the JavaScript file. So we didn't actually have to declare it in the HTML. and then we're also, getting this, importing this in which, you can see here, I have a comment that says, skip. so we're going to go ahead and create our new model. And we're going to do a local model check. This model will be downloaded from Huggy Face Hub. So that's why we've imported that here. we have, allow local models. We've set that to false just to be able to skip that. and now we're calling, we're going to have the pipeline API. We're going to specify our task, image to text, along with the model that we're using. And, And then here again, it's like uploading the image, getting, the file, creating a URL, and then what we're going to do is here's how we'll actually add it into the DOM. create like an image element, and then we're going to, have call this function called get test description. And so here we're actually, that image URL, we're actually going to put that into, we're going to send that to the function here. And when the pipe With the, like the pipe here, when it returns, the text, that's what we're going to get. So we have this out, generated text, and then we're actually going to display that text inside, like just below the image inside the app. So if I go back up and I just click on app, here, upload image. So I'm going to click that. I'm going to click a different image and have a different soccer player this time. So here we have Alice Morgan, celebrating, scoring a goal several years ago with Rosabelle. And here it has a description of two women are playing soccer on the field, which is a pretty great, it's a pretty, accurate description, I would say. And this particular model kind of has pretty basic, test descriptions. However, maybe there's another model. That's in the transformers, like library that I could convert that may be able to provide even more detail about this particular image. I know that also I've seen in the documentation that reinforcement learning. Is a supported task with, in Transformers js. maybe it's possible to be able to take a model, train it, kinda do some reinforcement learning to maybe kinda like work with, adding, like working with your own data set and creating, and like adding more onto that particular model. If you'd like to learn more about how you can work with TensorFlow. js or Transformers. js to develop your own AI applications, here's a link, and send my slides. And, and if you would like to be able to view these slides or even the code examples, you Feel free to scan this QR code and you'll be able to see, I'll actually have a link tree where you'll be able to go directly to the code examples, the demos and the slides as well. Thank you so much for watching this talk. I hope you've enjoyed it. I hope you've enjoyed the Conf 42 JS conference completely. I hope you've learned a lot. And again, remember, you don't have to be an expert to get started with working with AI in the browser. However, once you get going with these different types of libraries, such as TensorFlow. js and Transformer. js, you can definitely continue to build upon the knowledge that you gain. Thank you so much again, and have a great day.