Welcome to this Javascript session. Canvas pixel manipulation beauty comes slowly. This Conf 42 talk is basically a coding session, which I call pseudo live coding session because, well, it's pre recorded and it will deal with the Javascript canvas API and in the second part with the usage of web workers to ease user interactions when there's a lot of computations to do. So these are learnings I had on a personal project called Mosaics, which transforms a map into a color map with kind of mosaic delimited by roads. I'm Victor. I'm a lead developer at Theodo company based in Paris. Well, as for the Paris office. So where exactly in Paris is Theodo based? So you can have a look at this beautiful map on maposeic.com. Maybe you can recognize Paris map if I zoom a little bit. Well, Theodo is kind of here, so it takes a few seconds to compute it. We'll discuss it later, but after a few seconds we have a beautiful mosaic. Towns are beautiful, but such is the countryside, such as this town in Borgoin. This is actually the place where I had the id during the containment to draw automatically maps delimited by roads. So I was doing it by hand and that's how I looked at the canvas. So our goal today is to build this mosaic from scratch. In order to do so, I have a small project running on localhost, a small react app that displays a map on the left and on the right. We are going to progressively build our mosaic. So let's have a look at the code. I have a react app with typescript, and here is a component called canvas demo that renders on the left the mapbox container, and on the right the mosaic container that we're going to fill. So what exactly is Mapbox? Mapbox is a great library that allows you to build a maps. Just calling Mapbox map a new object that is initialized with the container. I pass the reference of the container a style. So here the style is a satellite maps, a zoom level between zero and 22, I think, and a center to initialize the map with. So here I put a random longitude latitude so that we are able to travel a little bit. And when the map is rendered, I call this unrender function that console log renders and stop the loading the loading indicator. So if I refresh, you can see that there are a couple of, couple of renders. Okay, so our first steps for pixels manipulation, we are going to first paint a rectangle, then access the rectangle pixels. We're going to paint the pixels randomly we're going to set the size to look like the map box size, copy its pixels, apply a transformation, and then we'll apply the area detection algorithm to paint the mosaics. So let's first add this rectangle. So I initialize a constant with the canvas element that I get by id. So this is a reference to this id, a canvas HTML element. Then I need to access its context. Okay, so there are a couple of different contexts available and I'm using the 2d context. And then the mosaic context is possibly null. So if it is null, let's just return. Okay. Then I say, okay, I want you to fill with the style color magenta and fill a rectangle at the initial position. So top left corner with the canvas dimensions. So how does it look? Well, we have a beautiful, very beautiful magenta rectangle with the canvas default size actually. So 300 by 150 pixels. Okay, so that's a good first step. Then we're going to look at the pixels because this is an image. So we should be able to see what this is composed of. So the mosaidata is actually the context on the context. I call this method guest image data with the top left corner and the size of the data I want. Okay, so if we console log it, what do we get? Okay, image data. So you see there is three properties and we're console logging the data directly. Okay, so we have an array of numbers, actually eight clamped numbers. So clamped means that they're between zero and 255. And actually each pixel is composed of four numbers, the RGB and a values. So that's what I put here. So our pixels are indexed from top left to bottom right. The first pixel is this one. And each pixel is composed of four numbers. Okay, so magenta is a mix between blue and red. Okay, so having that, let's paint the pixels. So we're going to paint them randomly. So this size is not defined canvas width and height canvas. And let's iterate over the data of the data. And at each pixel index I, we assign random pixel value, a random number between zero and 255. And then once we have this, we need to apply to the context, the new image data with the put image data method. Okay, so now you can see that we have kind of mosaic, actually a lot of pixels with random columns. And each time the map box renders, map renders, we have a new computation. So maybe I could just display the console here. Okay, now we want to set the same size as for the mapbox shape. So we need here to access the mapbox canvas. Okay, so the same way we're going to use a mapbox context, which this time will not be the 2d context, but instead the Webgl context because mapbox uses webgl API and we'll set our mosaics canvas width and height with the drawing buffer width and height of the mapbox context. Okay, so we'll need to do this before. And the map is actually the object we just constructed here. So we need to pass a maps to our function map gl dot map object. Okay, so the same way this context may not be it presence. Okay, so what do we got here? We got actually a huge canvas that is twice the desired size. This is because on retina screens mapbox renders twice as much as pixels per directions. So actually I need to fix the CSS size to access the CSS size here. So if I put a width of let's say 644, it's not going to make it. Okay, now we're good. So we need to do it in the code. So that's why I use this. So I use the style property of our HTML elements and I set it to a string necessarily with the size divided by two. So you see that the width of the canvas must not be confended with the style width property. Okay, so it started looking nice. What do we have to do now? So we are going to copy mapbox pixels. So in order to copy mapbox pixels we need to get Mapbox pixels. So actually we're going to use the Webgl API before or after this? Yeah, actually I don't need to console like this. Okay, so I construct a new int number array of the correct size. So width times height times four because each pixel has four coordinates. And to write the pixels in this array I need to call the read pixels method on the mapbox context with initial positions, the size and some other constants and also the reference to my constant. Okay, so now I have the mapbox pixels so I'm able to copy them. Okay, nice. So it's not actually done yet in Shemando Taranel because you see Shemanda Taranell here is written on the opposite side. This is because the mapbox, the Webgl convention started in the bottom left. So we need to apply some transformation to our pixels. So here I wrote some utils to be able to do that. So it considers that we have an x and y axis and each index position can allow us to find the x and y point position. So basically when we start to find a transformation from one canvas to the other, I have the I position in the index position in the mosaic reference, I convert it into an x and y point. Then this point in the mapbox canvas is just the same, but with a transformation on the y axis. And then with this calculus I get the map box index, pixel index. So I just need to call this little functional it. Okay, then I get the mapbox pixel index. So now we recreate, so that we could completely paint access the pixels of a map. So we are going to apply our detection algorithm. So let's describe a bit this detection algorithm. So the source is basically binary color image and the target is going to be a colorful image. So I have to detect three areas here in this image. So how do we do? We iterate over all pixels of the image and we make sure we delimit the contour of the image. So this is a zoom. Well, not actually a zoom, but yeah, with very big pixels our source. And we'll iterate from top left to bottom right with this eye index. And we are going to detect first this black area. So the source color is black and we want a target color of Xiong. And we call this routine paint area which push the index in a stack, in a two visit stack. And while this stack is not empty, it pops an element and paints this element. Then it looks at the neighbors of this element. So there is one at east, south, southwest and north, and the west and north pixel are not considered here. And if the neighbor color is of the same color, which is black, it pushes it in the two visit stack. So it will push here the south pixels. And once the stack is empty, Webgl have this first area painted. Then it takes the next pixel, well that we haven't visited. So each time we visit a pixel, we also mark it as visited and it paints the white area the same way. Then these two pixels have already been visited. So the next pixel will be this one and the target color this time will be burgundy, which is Bordeaux. And here we go. So I just need to call this transformer. Instead of doing this, I have a class canvas data transformer, which is actually the implementation of the previously described algorithm, which has, well, is contrasted with a source pixel array, a target pixel array, a size, and it generalizes a visited pixel sex. The paint target data is the main method and it will do what I previously described. Okay, for each area that we have not visited, it calls the pen current area method, which initialize this tag, et cetera. Okay, let's look for example at the method adjacent points. This is the method that gives the neighbor of a point. So for each point, each point has four neighbors, southeast, west and north. Okay, so having this, we'll call the transformer method, paint target data. And now what we have from our transformer, we can get the target pixel array. So actually put image data, receives an image data. So actually it's the image data that we need to pass here and we need to set. Actually we cannot set it like this. We cannot assign data to a value because it's a read only property. So we use the set method. Okay, let's look at it. It's working even with this shape, which is quite nice. So we have a threshold, although it doesn't work very well with full colored images because we just have a threshold to detect white areas. So that's why I'm using now another style for our Mapbox mile maps, which is a road style that I customized. So that's what is really nice with Mapbox is that there is an application tile studio or something. Yeah. Where you can customize your map styles. And this is the one I created to be able to render the map. So actually we're facing here, I'm facing some user interaction problems because all the computation is taken by my algorithm and it blocks the UI. So with difficulties here, I want to drag, but I can't get it. So even if I want to see the Kamarg, I'm going to have a lot of troubles to do this. So that leads me to the second part of this talk, which is how to ease user interaction with a web worker. So we want to run the process of the algorithm in a web worker. So why do we want this? Web workers are a simple means for web content to run scripts in background threads. That's perfect. The worker thread can perform tasks without interfering with the user interface. Okay, so basically, how does it work? Basically we initialize a worker with a file and we send on the worker side, we have an on message function that listens to messages, do the jobs, and then pass the result to the main thread. Okay, so the problem is for us is that we cannot use this for me, actually, I cannot use this directly because webpack is going to build the application and the URL is not going to be available anymore. So that's why I need to use a worker loader. So worker loader is a webpack module that I already installed in my node modules. Okay. So I just need to import the worker with this syntax. And actually this syntax is to make webpack understand to load the file with the worker loader. So what I also had to do is to eject my create react app to be able to override webpack configuration. So that's what I did and this is what my config looks like. So I'm using customize. Correct. But it's basically the same syntax as the one we just saw. Okay, so I'll import. Okay, we also have in this documentation a typescript hint that indicates you to declare a new module so that web worker is understood by typescript. Okay, so this is where my custom typing is. I have here a paintworker that initialize. Well, there's a constant that we call the on message method on. And when we receive a message, I will console log the message. So new worker. And the URL is it. So our worker is paint worker. Okay. And const worker, new worker. That's what we're supposed to do. Yeah, you'll need this because there's a rule added by create react app that doesn't want you to use this syntax because they don't want to be bound to webpacks actually. So that's why to ignore next line. Okay, is there any log? No, because we haven't passed any message to our walker. So before we do all this transformation, we'll pass a message. Work on this please. And Webgl listen to the response. The worker received the message, which is a message event with the data. Work on this piece and in the other side we receive the response. Got it? Okay, so now we're going to be able to put this in our walker so it's not the maps box pixels, so work on our data. We're going to initialize it with the data we want. Okay. And this is actually an object with the type I created and. Yeah, sorry, worker payload. Yeah, worker payload is an object with what I need here. And we're going to use it in our class, in our constructor, so there is no event anymore. Okay. And then when I call the transformer paint target data, I will pass the data that was just painted. Okay. And here I need, instead of doing this, I'm going to call the worker with the payload and target pixel array will be the mosaic data, the data. The source pixel array will be the walker, the mapbox pixels and the size of the canvas. And we'll post this payload and on the reception so it's the same data which is a worker response type UIT array I can console receives. And then we'll set the Masai data to this new data we just received and then put image data, the new image data and set is loading inside the on message function. Okay, reorder to type our magenta pixel and you see that the UI is not blocked anymore. However, there is a lot of render that are triggered. The worker previously is doing jobs that are outdated. So to prevent this I can terminate the worker job each time I'm about to do a new render. So this aborts the worker and it's also killed it. So I need to recreate it. That's why I want it to be a variable. Okay, and now you see that the worker is not doing any unnecessary renders. This is the only message response I received. Okay, so now we are done with our beautiful mosaic. As for a conclusion, as you can see, my solution is quite slow, so the algorithm is quite long to compute. So for example, for the 5 million pixels I have on my full screen, it takes up to 5 seconds, so it's very long indeed. So how could I improve this? So first I asked myself, why does mapbox map render so quickly? So first they apply only on layers, so they have a background, for example. Then they apply a layer of road, then a layer of forest, so they don't need to fill the whole image pixel by pixel. And then they use the Webgl API which allows you to draw shapes very efficiently thanks to vertex arrays. So maybe I could inspire with this with a new algorithm. Algorithm that would follow the road with the GPS coordinates and find the intersections to delimit the area's vertexes and then apply a layer with these vertexes. Another, another option. I asked myself why a software like Photoshops is so fast to detect areas. And one hypothesis is that it uses all the capabilities of the operating system. So I would like to try to run my process with binary instructions which can be compiled with webassembly and maybe it could improve. Well, it would be interesting to see the gain with this, so if you have any suggestion, don't hesitate to contact me. So I put my email address here. Thank, thanks a lot for your attention and I hope you'll do some great drawing with the canvas API.