Hello everyone, and welcome to my talk. The enters and exits of Python's context managers. My name is Mason Egger and I serve the developer community at Digitalocean. I'm currently a developer advocate there focusing on DevOps cloud infrastructure and Python. You can follow me on Twitter at Mason Egger, and if you have any questions about this talk or anything else, feel free to tweet at me or email me at Mason at dot co. So let's go ahead and get started. So who's been writing some Python code and has seen something like this where they need to open a file and you use the with open file statement, give it a variable or give it a value file permission to open and then assign it to a variable and then create a new variable, maybe called text or input, and say assign that to the value of the read. So essentially what this code is doing here is we're reading text out of a file into a variable. This is probably something that most people here today have seen, and this is actually a really cool tool in Python that a lot of people don't really talk about. And it's called the context manager. It's also lovingly called the width block. It's what I've called it forever until I figured out what they were called. But yes, if you ever hear someone talk about the width block, they're talking about context managers. Context managers essentially are a tool built into Python that guarantee that some operation is performed after a block of code, even in the case of exception, return or exit and things was basically designed as kind of a way to give you that try, finally pattern that we're used to seeing in other programming languages, maybe say Java, for example. So it was designed to give us this feeling and give us a simpler way of implementing the try finally, without all of that excess try indent over, except all of that. So that's what it was used for. And it essentially allows us to have reusability. It results in cleaner code and it's considered more pythonic. So context managers and creating context managers is very much encouraged by the Python creators, by the Python community, and it does result in some very well clean written code, and it just makes it really fun to work with. So let's talk about context managers. What do we use them for? What is the purpose of a context manager? Context managers essentially allow us to spin up and tear down resources in a safe way. So we know that whenever we open that file with the width open, we know that it's going to be closed at the end of it. Once it leaves that scope of that function of that variable, and that's actually where a lot of the value comes in, is that we know that we can spin them up and tear them down without any exceptions stopping our code. So they're very useful for file management. They're also really useful for socket connections whenever you need to connect to an outbound socket. I believe the Python networking libraries use context managers all over the place. They're also good for things like database connections where you need to be able to establish and disconnect from the connection, but they're also used in other things like game environments. If you've ever seen the Python gaming library engine, PPB, it's basically essentially one giant context manager. It's a really cool project, but basically whenever you start up a game, you're like, hey, I want there to be a grid or a display. Let's go ahead and open that up. And whenever we're done, whenever we exit that code base, let's exits out, let's tear it all down. And that's essentially why it uses a context manager. They're also really good for managing global state. They're good for locking, mocking and testing. They're very big in mocking and testing. So that way you can change something really quickly and then change it back, logging and so much more. There's an infinite amount of things that you could use context managers for. It's really only just limited to your imagination. So let's talk about how you actually build a context manager. Now, there's multiple implementations. We're going to talk about a few of them today. The first one we're going to talk about is the context manager class implementation. This is where you implement your context manager as a specific class, and then you would import it into your code to be used. Basically you implement this by using what are known as the magic methods, which are methods that are by default in a class and you can override them to use them. And basically we're going to use the enter and exit magic methods here to create a context manager. So if I was to create a context manager, I would say something like class my context manager, def Dunder enter self return here. Whenever we go into our context manager, let's just say, hey, I'm here, I have entered, and for now let's just pass on the exit. We implement pass, and it's got a couple more variables. We'll come back to that, but let's just pass on it for this section. So if we were to import this context manager, we would basically say with context managers as CM, after we've imported let's print hello and then print CM. And what we would get here is we would get the word hello because we are within the context managers. But we would also get here because this is the thing that is returned back up. So when we said in our context managers return here, the value of here was actually stored in the context manager variable CM. So whenever we actually printed CM, it was just a string. You could store anything you wanted here to maintain a kind of record of your context managers. I do believe like the open method puts the file handle there. So even though once you've exited the context manager, you will still have access to the file handle if you so choose. So the enter method is pretty interesting. It's executed at the very beginning before the width block is even entered. So before it even gets into the code of say, where we did our print statement, the enter method is executed. It's a magic method that only takes one argument, which is self, and it's used to establish connection, modify system, setup, et cetera. It is however a really good idea whenever you're using context managers and when you're creating your own context managers for sure to keep an original copy of anything system related that you modify so that you can change it back. So you could here modify and mock the standard out system. Now if you were to do that and not save it, then however you did your modification, it would be kind of stuck that way and you'd have no way of changing it back. So you want to make sure that if you are going to be modifying any critical things that you want to be able to put them back when you're done with them, it must return a value that is stored in the variable specified as as. So the magic method enter must return something. And whatever variable is specified in as, which, as we saw in our previous code was the variable CM, it will be stored there. So an example of this, let's create a class called, we're going to call it yelling text. But basically what we're going to do here is we're going to take over standard out and everything that we do is going to come out in all caps. So we implement our class, class yelling text and we implement def dunder enter self. So we're going to import sys so we can modify the standard out. We're going to say self standard out equals sys, standard out write. So we're going to just basically have that there. So that way we have it saved for us and then we're going to say SYs standard out write equals self yell and then return yelling. That's the variable that we're going to return back up to the context manager. We then say def yell is SYs standard out text upper. So basically we're just going to say we're going to set everything to uppercase. And whenever someone calls SYs standard out write, it's going to actually call the yell method instead, which is going to just uppercase all the text. And then again, we're not going to do anything with the exit method yet. We're going to get to that in a couple of slides. For example, if we import this in and then we say with yelling text, we said print hello, and hello should show up in all caps. But as a context managers, we expect for it not to continue yelling after the fact. So what we would like for it to do is that the I should not be yelling should not be in all caps. Our output, however, is hello in all caps. But as you can see, the I should not be yelling is still in all caps. So we've kind of messed some stuff up here. Now, if we were to continue with our code, all of our standard out from here on out, everything that was sent, the SYs standard out write would be in all caps. And we definitely don't want that. So now we should talk about the exit context managers method. The exit context manager method is executed after the body of the width statement. So after all the code within the width statement has been executed, this is kind of the cleanup. This is what's going to put everything back the way it needs to be. It's going to close. File handles close connections. So this has happened after all of the code within the indentated block is executed, this method returns a boolean flag indicating if there is any exception that should be occurred or suppressed. So maybe we don't want to actually raise the exception. So maybe we're using a file and we go to close it. The with block goes to close it and the file doesn't close. This is kind of a problem, but is this bigger than a problem than the entire program crashing? Do we want this little method or this little error that surfaces to basically stop execution? And that's where we get to choose. Like maybe we handled it in a certain way, maybe they put in some bad input, but we don't really want to do anything. Maybe we can log it, we can add some logging information in there, we can do some print statements, but we don't want this exception to actually crash the whole program. So whenever this method returns true, it says that the exception is suppressed if there was any otherwise, the exception will continue to propagate up until it's either caught or it crashes the entire program. So this method takes three different arguments, exectype, exec, val, and traceback. And these are essentially the exec type is the exception class. So it's the class of the exception that was raised. The exec val is the specific instance of the exception that was raised. So whereas the class is a little bit broader, the exception instance actually might have some data inside of it. Sometimes some of the parameters that are passed in via the constructor or whatever can be found in execval arg. So you can kind of dig deep in here to find the exceptions if you need to. It's not necessary, but you totally could if you wanted to. You'll know when it's right for you. And then traceback is basically a traceback object, and this is just the python traceback that was brought up. And if you could log this somewhere so you could see what's going on program running, but know that hey, something did break. So yes, this is the exit method, and basically it's used to put everything back the way we need it. So an example of this would be let's go back to our yelling text example. What we can do here is we can say in our exit we keep the enter and the yell the same. So we still are doing the yelling, we're still sys selfstandard text upper. But in the exit we import sys again to make sure if something were to happen, for some reason the exit method was called before the enters method. And since we're not importing at the global space in the global namespace, we need to import it again. I don't know why you would call the exits before the enter, but it could happen. The good thing is that re importing libraries in Python has like almost zero performance hit, because it's just a cache hit. So it shouldn't be a problem. Don't really worry about it. But what we do is here is we take Sys standard out write, and we set it back to self standard out, which is the standard standard out, which is the one that we saved up above. If we had not done self standard out equals Sys standard out write as the second line in the enter method, we would not have been able to set this back. So if we use the exact same example where hello, I'm yelling text print, hello, I should not be yelling. We do get exactly what we were expecting, which is that the text is uppercase for hello, and the text would be lowercase for the I should not be yelling part. There are exceptions in the exit method, though. So what do we do with an exception? So if it's true, and this is a statement from the previous slide, if true is returned, the exception will be suppressed. Otherwise, the exception will continue to be propagated up. So let's look at it like this. If we take our exact same one and we say, if exec type is exception, we print that there was an exception and we return true. So this is us like handling the exception. And by us returning true, it says, hey, there was one. It did come through, but we handled it. So if we look at implementation code of this, if after we print hello, we just raised a generic exception. If we had not done this and we had not handled it and returned true inside of our code, then it would have continued to propagate all the way up and crash the program. But we did say, hey, if the exec type is of the class exception. So if we see like any exception class, we'll say there was an exception, we're going to print it. And as you see here, it was printed, and then it continued and said, I was not yelling. So this is exception is really wide. It's a very broad one. Like it's the parent class of all of them. So it's a little bit weird to use it here. But you could do this with very specific exceptions. If there is a file not found exception, then maybe we want to handle it differently. Maybe we just don't return something or we do something differently. We don't set things back up the way we want, but we can handle the exception in there. So yeah, that allows us to handle it, and then we return up true and we're good. So let's talk about context managers that take parameters, because as I remember from our previous slide, when I said in def dunder enter, it can't take a parameter like self. That's the only parameter that can take. So how do we do something like the with opened block that we see where we're obviously passing in parameters to our context managers? Like, how do we do that? The with context calls in it first. So if you want to pass in parameters, it actually needs to be parameters into the init function, the dunder init function of your class. And then what it does is when the with call happens, it will call the init. If the in exceeds, then it proceeds to go through the inner code block and exit. So you'll need to write a constructor if you want to take these parameters. They don't go as the enter parameters, they just have to be as redundant in it. So let's write a context manager that takes parameters. And actually let's go ahead and just re implement the with open one that we're so accustomed to using. Let's call it class file open. This is going to be basically almost the exact same as what we're opening up, but let's say self dot name self dot mode and then self file handle. These are going to be in our constructor. So we're going to give it the name which is like myfile txt in the mode read like rrb rw, the standard Unix file modes that we typically use. And then we're going to take those in the init. And then for the dunder enter we say self file handle equals open, self name self dot mode. So this is the same opened that we're typically used to seeing. And this is what we would write if we were writing regular code. And then we're going to return that file handle so that way the person can actually use it. And then when we exit we're going to just close that file handle. And then if we import it in and we say with fileopen file text read as file handle text equals fh read. Ours looks almost identical to the one in the very beginning, except it says file opened instead of open because it's our own personal class name and everything works exactly the way you would expect it to. So that's how you implement context managers as a class. But there is one other way that you can do it. If you don't want to use a class, you can implement them using context lib as functions, with a function implementation. So these can be created by using generators, by using the yield function and decorators. So the context library and the contextlib context manager decorator allows you to specify a specific function as a context manager. And then the way that you specify what is the enter and what is the exit is the yield. So you use the yield function to separate them. So if you remember, like we have our enter and then we would return something and then we would exit, that return is basically the yield. We're yielding it back up and then we're waiting and then we go through. So context managers functions that are using this. So let's go ahead and do one. If we have a whisper function, which is basically everything that is said is going to be the opposite of yell. It's going to be quieter. Then let's go ahead and just decorate this with at Contextlib. Context manager, we had to import the context library. So we import sys. We say original write equals sys standard out, right? We're saving that original one so we can set it back up. We say def whisper write text. The text that comes in is original write text lower. So we're just going to lowercase everything. Then we say sys standard out write is equal to whisper write. So this is all what would go inside of our enter dunder enters class, and then we yield, whisper. And then this whisper is actually what's going to be put into that variable. So we've yielded. Now we are done with the intersection, and then when we're done within the block, it's going to come back in and it's going to basically resume after the yield, and it's going to set the original right back. So if we do that, we say print, this should be whispered in all caps, and then print things should not. This is not whispered. And as we see that it did do it, we were able to lowercase the code that was inside the context manager and then set everything back the way we expected to. So context managers can also be implemented with the function just using a try, finally. So by doing a try, finally, this allows you to handle the exceptions. So it's our same function. We're still decorating, we're still setting up whisper right, saving the old one. But what we do is the first thing we try to do is yield whisper. Now, if any code inside of our block throws an exception, it will be caught by this try. And when that happens, if we raise an exception, we can catch that exception and then handle that exception. And then finally when we do our finally block, that allows us to set everything back to being right. So instead of where with our other class implementation, we returned true and it was handled. Otherwise, we're basically using the try, except finally block here to handle the exception ourself without having to have that class. If we do the exact same thing, this should be whispered. We're going to raise an exception. And then we say, this is not whispered. Essentially what happens is this should be whispered. Now look, this is interesting. The exception inside of my block is exception happened, but as you can see with a capital e. But this one right here is lowercase. What happened here is we have not set things back to the way they were, so it isn't set back yet. And then we go to our finally this is not whispered and then we're good. So I hope you enjoyed this talk today. This is an overview of context managers. Really, this is actually about all there is to the base of the context manager. What you choose to do with them can be as complicated as you want, but with a context manager, the only things you really have to worry about are the enter the exit and handling of the exceptions. It's not a very complex python feature, but is definitely a very powerful one that is really useful and I hope that you get to play around with them, you get to have some fun with them, and I hope that you'll start to incorporate them into your python code. Every example that's on this presentation is on GitHub, so you can download at Mason context managers sample code and you will be able to run all of my examples, play around with them, change them all you want. Follow me on Twitter mason Egger if you want to hear more from me, if you want to see when I'm doing other presentations, if you have any questions and you want to tweet at me, please feel free. I love answering everyone's questions and if you're looking for an ebook, something to learn Python with. If you're new to Python, Digitalocean has a how to code in Python ebook that is amazing. That was written by some of our amazing developer educators and you go to dot co slash ebook Python and you can get a free book that teaches you how to do Python. Thank you very much for your time today. Thank you very much for attending my talk and I hope you enjoy the rest of the conference.