Hi, my name is Ishuah Kariuki and welcome to my talk in this session. Today we're Golang to cover terminal emulator basics in Golang. A bit about myself I am the principal backend engineer at hover. The mission at hover is to create an inclusive Internet. This entails creating technology and services that enable developers in emerging markets to actually build for the community around them. I am also a triathlete in active training. Makes most of my mornings miserable, but also goal oriented. And finally, I am an untenable troublemaker. First thing we're going to deal with is the Linux Tty subsystem. The Linux tty has a very rich history and I'm glad for this opportunity to talk about it. TTY stands for teletype, and the definition of the word teletype is a printing device resembling a typewriter that is used to send and receive telephonic signals. This is the image you get to see when you google the word teletype. It was mostly used in the early to mid 20th century and was developed as an improvement on telegraphs. So how did this typewriter looking device end up being an essential part of the Linux operating system? So earlier computers could only run one program at a time. But in the 1960s computers were powerful enough to interact with users in real time. So pragmatic engineers at the time thought about the situation and instead of creating new input output machines, they reused teletypes as input output machines for computers to enhance the real time interaction of users with computers. The main reasons were teletypes already in the market. People knew how to use them and technically they fit the bill. You could input by a keyboard and output by a printing paper. The image you see on the right is an output from the game. The Oregon trail. The very first version I'm pretty sure you've heard of, or you played the game at least, the Oregon trail. But the first version of the Oregon Trail was created in 1971 and it ran on a mainframe computer and user interaction was via teletype. This is actual output from the game on a teletype. So as technology improved, video terminals replaced hard copy terminals as paper terminals. And the teletype that you see here, the terminals that you see here is the VT 100. It's one of the most popular ones and it's also one of the first terminals to support unseen escape codes. Focuser control now we get to see how physical terminals is connected to computer. From the image. From diagram you can see that the terminal has a direct connection to the computer via a UART module UART stands for universal asynchronous receiver and transmitter, and the main purpose is to receive and transmit serial data. Now, from the diagram above, you can see now the TTY device consists of the UART driver, line discipline, and the TTY driver. Let's dig a bit into this component. The UART driver manages the transmission of bytes between the teletype via UART and line discipline. What it does is transfer keystrokes from UART to the line discipline. It also sends back anything that's sent from the line discipline back to the teletype. So that's it. That's basically command execution, sends characters in and characters out. The second part is a line discipline, and the line discipline maintains a character buffer. This is anything that's typed on the teletype is buffered on the line discipline until the user prints enter and the characters are sent forward to the TTY driver. Line discipline also handles special editing events. Backspace, enter, clear line, and then final thing, it echoes back, keystrokes back to the URT driver so that the user can see what they're typing. Then the final component is the TTY driver, which sends characters to the foreground process. Any program that's waiting on the foreground will receive user inputs. Most of the time you'll find that that program is either shell or a subprocess of the shell. This model, the TTY device, the whole of the TGI device resides in the kernel. So again, technology improved and processing power really improved on computers. Also, computers shrank in size, so there was really no longer a need to have physical terminals connected to the computer. Instead of having physical terminals replaced with software terminals, software emulated terminals. That's where the term terminal emulators comes from. A point of note is that terminal emulator basics not be confused with a shell, and we're coming to that in a bit. Terminal emulators just work in the same way as their physical counterparts, with the biggest difference being that there's no uart connection, there's no physical Linux between terminal emulator and the line discipline. So again, in this model as well, the terminal emulators exists in the kernel. One of the biggest constraints with having the terminal emulators in the kernel is that it was a rigid design. Programmers could not build on top terminal emulators basics since it resides in the kernel. An idea came about where the terminal emulator should be moved from the kernel and maintain the rest of the TtY subsystem still in the kernel. So line discipline, session management still remains in the kernel and then move terminal emulator basics user land and that's where we have pseudo terminals. That's how we have pseudo terminals. It's an improvement on the existing design. So terminal emulation moves into understand and the Tty subsystem remains in the kernel. This diagram explains how the pseudo terminal works. Pseudo terminal consists of two files, the Pty master and the Pty slave. The Pty master is attached to the terminal emulators and the Pty slave is attached to the foreground program or the program that you want to control. A note about shells. A shell is a program that resides in user land and manages user computer interactions. A good example is z shell. Fish bash, you're all familiar with this. So yeah, this is the basic history of terminal emulator basics. Why we have pseudo terminals. The next part we're going to dig deep into how to create a terminal with Golan. Cool. So this is a very simple program that draws a UI with awards. Hello, conf Golan 2021. When you run this program, this is what you get, just a simple print. This is a user interface, a very simple user interface that is by the end of this code lab will be somewhat interactive. Terminal a basic terminal that's interactive. So the first thing we're going to do is connect to the pseudo terminal. The pseudo terminal we're going to use a package called Pty. It's an amazing package. It's very intuitive to use and we're just going to dig in. There's really no introduction to a codelab. You get into it and start with my imports. So this is what I'm going to use. Going to create a command execution and I'm going to use the time package. And then let's import the Pty package creek Pty GitHub. It's very intuitive. The documentation is amazing. If you want to work with pseudo terminals you can check it out if you haven't used it before. Cool. So first things first, let's create a command execution with the bash. The shell create a command execution of the shell so that we can actually connect the pseudo terminal slave to the shell and get a reference to the pseudo terminal master. Let's start with this. If I sound a bit out of breath just from my evening run, so do not hold it against me. Bear with me. That's what best time to use. Ben Bash. This is a path to the actual executable bash. And this line creates a command execution command struct that specifies we want to execute the bash, we want to execute bash. Cool. And then the next thing we're going to do is now we're going to assign a Pty slave to this command execution. Like so. This function will assign the Pty slave to the command execution of the bash. In simple terms, we're going to attach the bash to the pty slip and this function returns reference to the Pty master. That's what we're going to call P. P is now the Pty master. And if you can remember, pseudo terminal does contain two files, the Pty slave and the Pty master. Pty slave is connected to a process you want to control, which is now bash. And then Ptymaster is attached to the Terminal emulator and that's what we're going to use every single time we want to run a command execution. Let's just check if this fails. If that fails, what do we do? We red quit, we say log the error could not open Pty and then we're going to exit infrastructure. And finally we want to make sure that when we quit, when we finish, when we terminate our terminal emulator, we also terminate, we as well terminate the bash execution struct that we created earlier. Cool. That's that. So we have this, we have P, which is now P is tty master. So we have P. What do we do with Ptymaster? You send commands to it. So let's just write a simple command, a very simple command, right, sorry, bytes. So this sends a command lf and the return courage, return courage is a presenter. This tells the line execution that, okay, we are ready to send this command to the foreground process and let's do this, let's wait for a second and then read from the Pty master. So we do that, make, create a slice of bytes, we read from the Pty master into this slice of bytes and check if there's an error. New, we do have an mention that we can't read and yeah, last thing we want to do is this is what sets the text that you see on the Ui. Let's remove this and say what do we want to print? We want to print a string of the slice of bytes, convert the slice of bytes to a string and then print it to the Ui. That's it. So what happens when we execute this program? This is what we get. You can see the command ls and the return carries, which has unfortunately been also printed. And then that's the only file in that directory. Boom, that's it. So we have a connection to the pseudo terminals we can send commands to pseudo terminals. At this point we haven't attached the keyboard yet, so that should be our next step to make it interactive in real time. So next step, connecting, reading from the keyboard, the better way of putting it. When reading from the keyboard, we need two callbacks. One that reads, that listens on events on special keys, specifically the enter button. When the enter button is pressed, we know we're ready to execute the command. Send the command to the foreground process and see what happens. The second callback function is going to read runes. This is any other character, any other key that's pressed on the keyboard that's not a special key, any other key that's not either, enter shift, backspace, deletes, arrow keys, all that. So that's going to also, we're going to listen for all those key events. Anything that's pressed, we are going to write it directly to the pseudo terminal master. Boom. Let's start. Start with special key. Press callback. It's going to be called untyped rune. Sorry, untyped key. Fine key. The event. This also relies on the fine. That's what I'm using to draw the user interface. Sorry if I did not mention this before. Fine ui toolkit. It's amazing. If you haven't used it before, you can check it out, tinker with it. If you're building any front end user interface and you haven't bumped into this yet, please tty it out. Sorry, it's meant to be that. Okay, here, what do we do? We write, we write to the pseudo terminal master and we say we are ready to executed. Cool. That's the return carriage tells the line discipline that we are ready to send this to the foreground process, toi driver and to the foreground process. Second, callback function is character callback. Going to call this untyped rune and it's going to do this. Take that rune and just send it directly to the pseudo terminals muscle. We're converting it to a string and send it to the pseudo terminals master as a string. So we need to bind these two callback functions so that anytime a key is pressed, this callback functions are called so on. Type key on type key. Okay, now we've set callback functions. Anytime you type on the keyboard, it's going to type anything. Press enter. That command or whatever string you've typed is going to be sent to the pseudo terminals master, to the line discipline and tty driver into the program process. Now we also need to see what we've printed. So let's write a very simple go routine that's going to refresh the Ui. So Ui. So this is function and then sorry, we're going to sleep for 1 second it, sleep for 1 second it. And then make a slice of bytes. Let's make a slice of two wondered and 56 bytes, then read from the Ptymaster error equal and then we log that error you it's log it's. And then just update the Ui. Convert the slice of bytes to a string and update the UI. And of course make sure this is called and then write it Linux 64 I have something. Sorry, this is meant to be. Anyway, we're not doing this anymore, but ideally this should be. And remove this to use this. We're Golang to use this. Think we're good now, are we? Yes, should be good. So cool. So when you execute, when you run this program, you're going to get this. And this should be slightly more interactive than what we had before. Yes, you can see the output, but it's mangled. It doesn't make sense because you are basically waiting. The go routine is sleeping for a second. Let's go back to this, probably sleeping for 1 second, reading everything and then setting the updating the UI so it's chaotic, but it sort of works. Now let's try running a program that will stick to the foreground, let's say ping h. Yeah, UI is updating, but it's not making sense. There's no command history. We can't see a buffer of commands, of outputs. We're only seeing the last line that was written to the PTy, to the sudo Terminal master. And yeah, it's just chaotic. So let's try and make a more legible user interface. Let's print better to the screen. That's the next step. So in order to get this working correctly, we're going to do two things. We're going to have two go routines. We're going to update this go routine that refreshes the UI, and we are also going to create a new go routine that reads from the properly reads and puts everything in a buffer. First of all, you're going to have a buffer and the buffer size. Let's say we're going to have a buffer size of ten and that buffer is going to be updated every single time. So when we get to a buffer size of ten, if it's at the max size, the first item at the top, the item at the top is going to be popped and then we're going to append the new line at the bottom. So this buffer is going to be updated. So it's going to look much better, much nicer, and we're going to have, it's going to be more orderly semblance of orderliness, if that makes sense. Awesome. So let's start with this first go routine. Let's ignore that and have this read Rome Ptymaster function. Sorry then. Now. Cool. What do we do? Before we create that, let's declare two variables. The buffer, which is going to be a slice of a slice of room, of slices of runes. Let me just explain this. So it's a slice of a slice of runes. Each slice of runes represents a line in the terminal output. So every time there's a new line character, we're going to move to a new line and then happens all the new rooms that we read to that new line. Then let's have a better reader. Let's have a more orderly reader. We're going to use buffyo new reader. And this is, we are reading from the Pty master. Let me also take the time to update my imports and just make sure I have all this looks good. Cool. Let's go back to it. So once again, this is the buffer. This is what we're going to use to maintain a history of lines of output on our user interface, on our terminal. This is going to hold the history of command outputs and the reader is a much better reader. We're going to read runes now from the Pty master. Okay. So first thing let's happens a line to our buffer. Buffer is initialized. It's empty. This just adds one line, an pty line as well. It's just empty. So this is a loop that does a lot of the heavy lifting. All of the heavy lifting actually read rune. This just reads a rune from the Pty. Reads a rune. If, because you have an error and if that error is ten file, then we stop, we just return. Otherwise we exit. Now, if it's an end of file error, we return and then regardless, whichever error we receive, we're going to exit. Cool. Then the next step is to append the line, the rune that we just read to the current active line and then updates the buffer with this new line updates the buffer index. We know this is the active buffer index because it's the length of the buffer. It's the last available element in the buffer. So, for instance, if this is a brand new buffer and this is the first line, we're going to update that first line to this line that we've just appended the room thread and then. Okay, I'm using spacefam. Spacefam is amazing. It might make you lazy, but it's awesome. We just checking if the rune that we just read. Check if the rune that you just read is a new line character. If it is, then we move to a new line on the buffer. So first we check if the length of the buffer is greater than, let's say our maximum size is ten. If it's greater than ten, then we pop the last item in the buffer. The first line that we appended to the buffer is popped. That is if you've got into the maximum size of the buffer. Then you pop that first item. And then to that we append a new line that we just created a new line. And then we append that new line to the buffer. Cool. That's it then. The next part is now updating this go routine that renders to the screen. Let's reduce this time to 100 milliseconds and then let's clear everything, blow everything up, and then we're not going to use this line. Sorry, I am breaking so many vim rules right now. If you're a vim user, bear with me. Just bear with me. So instead of reading from. Instead of reading from pseudo terminal master, we created a go routine to do that above. So what we are going to do now is read from the buffer, our existing buffer. So let's say string explaining this in 1 second. Sorry, line range, we're workings through all the elements in the buffer. All the lines in the buffer, the slice of runes in the buffer, all the slices of workings in the buffer and then lines is going to be. We are just creating one long string from all the elements in the buffer and then setting this. Sorry. And then set that as we look to the buffer, append all the lines into one long string and then write that to the user interface. And that's it. Those are two go routines that we need. First is create a buffer. Read drones from the pseudo terminal master while we happens it to the append all these drones to the buffer, to a line and then all these lines to the buffer. And then second go routine reads everything from this buffer and then prints everything to the screen. It makes the interface a lot nicer than what we had before the chaos that we had before. So when you run this, this is what you get. And let's say type a command. There we have it. The user interface is a bit glitty because the refresh rate might be a little slower than the way we are reading from the pseudo terminal mass, the way we are reading from the buffer, and it's a bit glitchy. So let's say. Let's type something else. A new command. New command. Let's say cow. Say moon. There we have it. Let's do what you did before. Run a program that will stay in the foreground. Boom, boom, boom. And that's it. The user interface is refreshing. We can type commands from the keyboard. They're being executed as we expect. Foreground programs are remaining in the foreground. And this is a basic terminal. This is a very rudimental terminal. Of course, elements that are missing. We're not interpreting ansi escape codes. We're not interpreting a lot of special key presses like tab backspace, delete the arrow keys. We're not interpreting signal interrupts. Control C, control Q, control Z. There's a lot more to build on top of this, but this is the basics. This is how you can start off with a terminal emulator in go. Yes, and that has been my time. Thank you so much for having me. It was a joy having this talk. My handle is at issuer. Understand on Twitter. Reach out if you can. If you have any questions or any comments about this talk, best way to reach out to me is via Twitter. Thank you so much.