Martin McBride, 2020-05-25
Tags with context manager
Categories python language intermediate python
Programs often do things that need to be undone. You might open a file, database connection or network connection that needs to be closed afterwards so that other programs can use it. You might grab a large block of memory that needs to be freed up afterwards.
It isn't just about resources. Parts of our program might need to set a particular processing context - for example it might need to change the default mathematical precision, or catch runtime warnings. These states need to be undone when you are finished, otherwise the rest of the code might not function exactly as expected.
It is possible to undo these states manually, but there are dangers to this. If the code throws an exception, the code block might exit without cleaning up properly. If the code is very complex, with multiple exit points, you might miss a case, and exit without restoring the initial state.
With statements provide a neat way of handing this. When you execute a with statement, it creates a context manager that creates the new context. When your code exits the body of the with statement, for any reason, the context manager is called again to clean up.
In this article we will look at the most common use, ensuring that files are closed correctly after use.
Manually closing files
The most commonly used system resources are probably file handles. Every time you open a file in Python, it uses grabs file handle to tell the system that it is using the file. When Python closes the file, it gives that handle back. Here is some typical code that does this:
f = open('text.txt', 'r') # Grabs a file handle s = f.read() f.close() # Gives the handle back
File handles are global resources, shared by every program running on the system, and there are only a finite number of them. Modern computers have quite a lot of file handles, but even so it is good practice to always close a file when you have finished with it.
In addition, if you fail to close a file when you have finished with it, it can cause problems if other programs need to access the same file.
Manually closing the file, similar to the code above, works most of the time. But there are cases when it can fail:
- If the code throws an exception (for example, if the file read operation failed), the code would jump straight to the exception handler and
closewould never be called. Note that this would happen with any kind of exception, not just a file exception. If there was extra code in that block that threw a divide by zero exception, it could still prevent
closebeing called, which could cause problems.
- In more complex code, there might be more than one way to exit the code block - for example there might be return statements or break statements. It can be difficult to be certain that every possible case has been handled, and the code will always call
This is where a with statement is useful.
Using with statements
In the example above, we could use a with statement, like this:
with open('text.txt', 'r') as f: s = f.read()
The with statement calls
open and assigns the returned file object to the variable
f - just like our previous open call in the original code.
When Python exits the with block for any reason the file
f will get closed. this happens if there is an exception, a return, a break, or if the code block just ends normally. In every case,
close will be called.
Notice that we don't even need to call
close explicitly in our code (we could, but it would be pointless). We just let the with block do its thing.
We can nest with blocks, for example to do a file copy:
with open('text.txt', 'r') as fin: with open('out.txt', 'w') as fout: s = fin.read() fout.write(s)
When the inner block ends, both the files will be closed. In addition, if the output file failed to open, the input file would still be closed when its with block terminated.
An alternative way to create a nested with loop is this:
with open('text.txt', 'r') as fin, open('out.txt', 'w') as fout: s = fin.read() fout.write(s)
Which form you choose is a matter of personal choice, although of course if it results in excessive line lengths that should usually be avoided.
The context manager
A with statement operates with a context manager:
with contextmanager as x: with_block
A context manager is an object that implements
__exit__ methods. The with statement calls the enter method as it enters the with code block, then calls the exit method when control leaves the with code block for any reason.
The file example works because a Python File object is a context manager. In particular, it has and exit method that closes the file.
A later article will cover other scenarios.