Hey everybody, my name is Malisha McGregor and today I'm going to talk to you about making VR more interesting with JavaScript and brainjs. So before we jump into that, I want to give you a little bit of background on me. Right now I am a developer advocate at Conducto and we're working on these really cool CI CD tool and data science pipeline tool that'll make it easier for you to get just all of your apps out, and it's not using YaML, so there's that. But if you're interested in more DevOps stuff or more VR and JavaScript and machine learning, you can also follow me on Twitter at flipped coding. So to kind of get things going, let's start with just a general overview of what we'll be talking about today. The first thing that we'll cover is just a little bit of background on virtual reality. Then we'll talk about making a VR app. So there will be a little bit of live ish coding so you can see kind of the approach you would take to make a real VR app. Then we'll jump back into the presentation and talk about a little bit of background on machine learning, neural networks in particular, and we'll talk about adding machine learning to these VR app. So we're actually going to apply the neural network to the VR app we're working on in this presentation. These we'll go over a few other considerations that usually get left out of most applications. And finally, we'll wrap up with just some key takeaways and things that I really hope you remember and take away from this. So to jump right into it, let's talk about a little bit of background on VR. As you all know, this is a relatively new area of tech. It's been heavily dependent on the progress of the hardware, so there aren't a lot of production code games out there, but there are more than you would think. It's just newer, so it's taken some time to get off the ground. But these applications have such a wide range of industries, they touch. So you start seeing VR in medicine, these, they use it to help train med school students. And of course you've seen it in video games. My personal favorite VR game is super Hot, so it's already been there. And then there are other forms of entertainment. So maybe one day we will all just plug into the Matrix, but we'll be okay with it. But what's so interesting about VR for developers in particular, is that this is a completely new career path. So if you've been wanting to do something a little bit more artistic, but you don't want to get into video games development. This might be a really good alternative because you'll still get to work with things like assets and different physics engines. So it's kind of a way to get into more of that artistic side of coding, but you're still using these exact same languages. And of course, as the hardware keeps improving, there's going to be a need for more developers, there's going to be a need for more applications for this kind of technology. So keep an eye on those Oculus riff headsets or whatever Google is making at the time, because as this tech keeps getting pushed forward on the hardware side, it's going to need the software to support it. Speaking of which, now is the time that we get into the code, and I'll kind of walk you through how we make a VR application with JavaScript, in particular with the Aframe library. So before we jump straight in, I want to give you an intro on this game. It was originally inspired by Super Hot, and then it took a completely different turn. Basically we're making a really fancy search and find game. Like there's objects hidden throughout this world and the player is supposed to go around and find them all. With that being said, let's go ahead and look at some HTML. I bet you thought we were going to jump straight into javascript, but no, not yet. So first thing we have to have this index HTML file because it is the root file for the project. And what we're doing here is first we're importing the Aframe library, so that'll give us access to all these methods and nice fancy parameters they have. Then we're importing another library that's completely optional. This is just a GitHub repo that some kind soul put together to make different environments for Aframe. So if you want to kind of play around with Aframe before you make these commitment in making an entire world for it, you might consider using this package in particular for just some test environments. So I have nothing to do with it, but it is a really cool library. And here you'll see we have a couple of custom scripts that we'll talk about in due time because they relate to some of the other things. But just to jump right into Aframe, inside of our body tag, we'll have these other tags that are just a tags, and they're all representative of different entities that are in Aframe. So every aframe application will have this, a scene. And if I minimize this, you see, these scene contains everything in our VR world. And what we do is we add different components to the world. The first thing we add would be the assets that we need to use throughout the world. So in this case, I'm just going to import a few textures, which really are just images that you apply to different entities in Aframe. So you see we got a wood texture, metal, Brickstone, blah. One thing, I threw this in here just so you all would get to see it. But you can add your own music to your VR app. It's pretty easy. It's just an audio tag and you import the file from whatever directory is in your project. This won't play because I figured you guys wouldn't want to hear some kind of game music over the entire talk. But the first thing you import in your scene would be your assets. And that's just so they're ready for when you start to use these later in this file. And we're not going to go through every HTML element, but I want to make sure you understand what's happening with the Aframe stuff, so we might get really close. The first thing we're going to look at is this a box. So this is just a primitive entity that gives us a box. We have a box in the world now, and it's got this id, it's got a height of three. But Aframe is awesome. So it's 3 meters. All of the units for height or position or things like that in aframe are measured in meters. Just so you know, and aren't confused on what three actually means. And then the rotation is how the object is originally configured in the world. In this case, there's no rotation around the x, there's 60 degree rotation around the y axis and a 45 degree rotation around the z axis. These to show you how we use our assets, we're applying this wooden texture to this box. So that means this image up here will get applied as the surface for this entity. So all that means is that the box is going to look like a wooden block. And then here you'll see the position. So this is just the initial position that the entity will be in when the VR app loads. And again, these are all in meters. So we are 37 meters in these positive x direction. It's 1 meter in the y direction, so that it's not just sitting on these ground. And these, we have it negative 37 meters in the z direction. So that gives us how our left, right, front, back positioning works. And then this is another attribute. So all of the aframe entities use attributes, and you won't see this one maybe all of the time, but it does help when you need movements that are just going on in the background. If it adds to the environment, don't just have things moving because it's cool. Make sure it's something the user will like or they'll get some use from. But in this case, all we're doing is making this block float up and down to show you what it's like. So we are animating the y position of the block, and we're moving it from its initial position to 2.2 meters. And basically it does this up down movement over a two second duration, and it does it indefinitely. So this block is just floating between one and 2.2 meters off the ground. So that's really all you need for entities in aframe. Then you can add your own custom attributes, which is where things get way more interesting. And we'll talk about this one in just a little bit. So there's more entities, blah. You've seen how those work. There's cylinders, there's a sphere, but there are few others that would be useful to know about. So this plane entity, you see, it's just an a plane. We can use this as like a sheet in the world. If you need to put up a message, which of course is what we're doing with this, or you need to separate things somehow, a plane might be a good choice. So you see, it has a height and a width of 2 meters, so it's fairly large, and that's just where the position is with respect to the origin of the world. And inside this plane we have the base aframe entity. So when you want to make something super custom, or you want to have fine grained control over how your entities work, you might want to just use the a entity tag. It lets you set everything just by using attributes, so there are no presets, but you can build any of the other primitive shapes. But in this case it's just some text telling the player what all they need to find throughout the world. That's it. And just to wrap up in this file, there are a couple of things that you really need to know about. The first thing is the environment. I know you remember me mentioning that particular package earlier, but you need an environment for your world, or else the user is just going to be in a white, empty box, basically. And this environment isn't anything special, it's just something that came from that package that we imported earlier. So you can set all of these attributes based on what we imported and take a look at that library. But probably the most important thing about your VR world is the camera. It's really hard for users to get around your world or interact with it if they don't understand what they're looking at. So you want to take into consideration these point of view that a user needs to interact with your VR app the most effectively. In this case, we have our camera positioned right behind these user, so when they see the screen, it just looks like they're looking right in front of them. And the way we kind of give them some guidance in letting them know they're looking in front of them is this geometry attribute. We use it to add just a little black ring to the middle of the screen that moves around with the camera. And now I do want to go ahead and show you all how it works, but let me make sure I'm running the right command. There's always that chance. Okay, so I think that should be good to go ahead. Okay, so it is running, which means I should be able to just pull it up. Got to get out of that. So now we'll just go to localhost 80 80. And this is what the load screen for aframe looks like. But give it a second and the world will just appear just like that. So this is our VR world. And if you see we have that plane with the text on it, we have this little ring in the middle that follows the user's camera. And this is what the environment we chose looks like. So it's pretty awesome. You can just run around. Oh, we actually found something sweet. So I'm not going to show you what that does until we talk about this. I know you noticed up here we had that found item attribute and all that does is call this file. So with aframe you're able to set just some custom actions and different things that happen when a user interacts with objects in your world. And what we've done, this should look pretty familiar, is we've just gone through and gotten these elements that are in the world, and basically when you find it, it will move that object back to kind of the home base. So once you click on it, let me show you real quick. So we'll just go up to the object, go over here and it's gone. So let's see if we can actually get back to the main part. I think this is right. This is a lot. Yeah. So there it is. You found the metal sphere. That's it. So it's back here, and you need to go find everything else. But basically that custom attribute lets you add different event listeners to update things in your world. And in this case, we just want to move everything we find back to the base. With that being said, we can get back in here because you've actually made a VR app at this point, anybody can go interact with that like you're done. But if you want to be fancy, you can add some machine learning to it. So just to give you a little bit of background on machine learning, we'll go over neural networks. A neural network is just an algorithm that's used to make predictions. Honestly, that's all of machine learning. It's algorithms that make predictions on data. So a neural network in particular is made of layers of nodes. So what you see here is we have a lot of inputs, which are also called features in machine learning, and each of those inputs has a particular weight. The weight assigned to the input determines how much of an influence that particular input has on the predicted output. And usually there are multiple nodes, which leads to deep learning. So if you have a neural network with more than one node, you're doing deep learning. That's really all it is. Of course, it gets complicated as you start layering more nodes, but overall the concept is the exact same. You have some inputs, these have weight values assigned to them that dictate how much they influence the output value. All of that goes through some kind of fancy algorithm, and then you get an output. So that's the gist of how neural networks work. And then we'll talk about the library that I chose to use for this particular app, and it is brainjs, which is awesome. And there are a few reasons that I chose this over tensorflow, because I know that's kind of the standard JavaScript machine learning library, but brainjs is way easier to use. If you understand how to work with objects, arrays and arrays of objects, you're fine. You can do machine learning with JavaScript and with brainjs. You don't need as much of a mathematical background in machine learning as you do with Tensorflow, because Tensorflow, I will give it that. It does handle some complex problems a little bit better, but the syntax that it uses, it just needs you to have way more background knowledge than you actually need to implement a machine learning model. And brainjs gets rid of all of that. Plus the tutorials are super straight to the point and easy to understand. I tried to learn how to use tensorflow, and I got pretty far with it. I made some models, but it's a challenge figuring out what a tensor is and going through these documentation and just understanding how everything works together and what parameters you're supposed to put in which methods. Brainjs just doesn't do that. Again, if you can work with objects, arrays and arrays of objects, you can do machine learning in JavaScript. So part of a machine learning project includes choosing the features that might arguably be the most important part because it's going to directly influence what prediction you give to a user. So when you're choosing features, think about what prediction you're trying to make. So what's going to add the most value for a user? And then think about how are you going to get this information? Are you going to buy data? Are you going to ask users for data? Are you just going to find a way to take it from them? How do you get the information you need to make these features? And then again, I can't stress how important this is. What exactly are you trying to predict with your information that's going to determine the entire model, of course, because you'll need to know what inputs you need to get the prediction you're looking for. And then this is a fun one, we do these really cool tech projects, but at the end of the day, kind of the only thing that matters is will a user care if we make these update? So you're going through and making this really fancy machine learning program and it does all of these cool things and then you deploy it to production and nobody says anything because they didn't notice, or they noticed, but they didn't care. Before you spend the time investing into building a machine learning model, make sure that the predictions that you're trying to get are things that people actually need or care about. So when you're choosing your features, just keep these things in mind. Oh boy. Now we get to go back to code and I'll show you how to integrate machine learning into a VR application. And all that is it's just going to be a node server back end with a few method calls and maybe one endpoint. That's it. To get started, you set it up kind of like you would any other Exprs app, no big difference, except we're importing the Brainjs library. Keep that in mind. So set up the Express app, use some stuff by some comments in case you want to go look at GitHub. These are all in there. But this is where it gets interesting, our initial training data. So what we're trying to predict with this machine learning model is whether a user will finish the game or not. And we make this prediction based on how they're progressing through the game. So we look at their current location in the world and we figure out if they've found an item or not. And we look at how many steps they've taken. Based on all of these inputs, we'll be able to predict whether they'll finish the game or not. So going into a little detail on these inputs, all this is an array of objects. That's it. We have our X, Y and Z location. For the player. We have a boolean value or just a discrete value for whether they found the item or not, or any item or not. And then just account of how many steps they've taken so far. So that is it for the training input data. That's it. And these, the training output data is honestly not much different. It's also an array of objects. And in this case it also has just a discrete value of whether a player will finish the game or not. The hardest part will be getting the data and then getting it into this format. Once you get here, everything else is pretty easy, which I'm about to show you. So first thing we need to do is to combine our training data into one array. And to do that we'll just use a quick map function and it'll return this object with both the input and the output. Then, probably one of the most fun parts is actually making our neural network model. As you can see, it's only one line. That's it. Brainjs makes it that easy. And I know it sounds like I'm advocating for brainjs really hard, but I have zero affiliation with it. I just really like this library, but all it does is you create a new instance of a neural network object. For this example, I've just given it three hidden layers, so that means it has three of those nodes we were talking about earlier. And this is the whole model. These is what a neural network looks like. So the next fun part is actually training these model, which is really cool because you just call a train method and you pass in this training set that we made up here, which is just an array of objects. That is all. So we have a method and can array of objects. Now we've trained the model and this will run and give you some stats on how many iterations it took to converge. What was the error at each iteration, things like that. So you get an idea of how accurate your model is and how efficient it is. Then in here we're just doing a quick test. It's just logged to the console so that we know it's working. But this is just the input data you would give it to get a prediction. So now that we have our trained model, we can go ahead and make our endpoint. So all this is is just an endpoint to get our user data. And based on these data, we return a new item location using our model. That's it. So we'll get their data from a front end request. Then we'll use the data to just make up some new positions and these we will return it. That's all. So that is what it looks like to use this model in a VR app, but the actual application of it comes in here. So we import this user model and this is just a call to the back end, like it's a post request. That's all that's in here. And it'll return where the item should be next. But this gets applied as another attribute on this item in particular. No special reason, just because I like that box these best. But anyways, so when that box has been found or it's been triggered in some way, this will get called and it will request that new position from the machine learning model. And that gets sent back to the front end and we just change the position of the objects accordingly. Basically, the goal of this machine learning algorithm is to move the objects around the world in such a way that encourages players to finish the game. So if they're not doing too great, then we'll move all of the objects a little bit closer. If they're doing really good, we'll challenge them some more and move all the objects away. And this is a really simple application for machine learning. You could get into bot design and just changing the entire world if you wanted to. This is just a quick little intro. So now we've added some machine learning to the app, and I hope you understand how that code works. And if you don't, just shoot me a message on Twitter and I'll send you the link to the GitHub repo. So now we can talk about some of the other things you need to consider. The first thing is the overall user experience. So when you're making a VR world, you want your users to feel immersed. You want it to feel like somewhere they are familiar with, somewhere that they should be. And to do that, you need to know what they expect. So that could mean things like working with the lighting or the camera position, or picking the right environment. Things like that have a huge effect on how users interact with your VR apps. And then probably my favorite but sadly, most overlooked area of all programming is just the ethical use of data. There have been a lot of questionable uses of data in the past, and when you get into things as personal as somebody's location and how they move around a room, that is some delicate information, and you can do some crazy things with that, whether you believe it or not. So however you're getting this data and however you're storing or using it, make sure that it's ethically sound so that you protect everybody out here. Choosing the best algorithm doesn't exist. If you're doing machine learning, just choose the algorithm that works best for your problem. Your data is going to be different. There's going to be different predictions you're trying to make. So you might have certain machines that only have cpus and no gpus that could limit the algorithms you're going to work with. The best algorithm is whatever's the best for your case, as long as it gives you a pretty decent accuracy. Your error rate is low and it doesn't take too long, you're doing a pretty good job. But if you're trying to do more serious machine learning, like you're trying to get into some heavy onboard hardware processing, you might consider looking at Python. I've worked with some of those libraries over the years for slam on autonomous cars and just like people, detection from camera data, things like that. So Python is exceptional for machine learning when you're starting to get into the heavier applications that Javascript just can't quite handle as well. So now we can wrap up, and I want to make sure you take a couple of things away from this. The first thing is that it's important to try to combine multiple areas of tech. Everybody tries to keep things siloed into different industries or into different programming professions. There's no need for that. Put it all together and see what happens. You'll be really surprised. These interesting things you can make when you start combining unconventional fields again, remember that ethics is important. Like use it when you're handling data, use it when you're writing code, use it when you're solving security issues. Ethics are important in tech, and they do tend to get overlooked sometimes. So just keep that in the middle of your mind and try new stuff. Like Aframe is such a cool library, but there's also a react 360 library that lets you make VR apps specifically in react. There's so many tools out there now that it's hard to say what's best or which one should be the standard. So just try out different things and see what works the best for your application. And these the most fun part of all of tech is just to learn things that you're interested in and things that help solve problems you're having. That's the best feeling. When you solve a personal problem that stems from something else you're interested in, it feels really good. So don't be afraid to learn really new, seemingly complicated things, because once you get in there and you start going through docs and you start writing a little bit of code, you get past hello world. It does get a lot easier, and that gives you so much more flexibility on what you can make. So I really hope that you were able to learn something from this talk, and maybe you'll go try making a VR app with machine learning. If you do, just show it to me sometime. Remember, you can find me on Twitter at flipped coding, so I'd love to hear from you if you have any comments about the talk, but other than that, I'm finished. I hope you have a good day.