With statements

By Martin McBride, 2020-05-25
Tags: with context manager
Categories: python language intermediate python

Programs often do things that need to be undone. You might open a file, database connection or network connection that needs to be closed afterwards so that other programs can use it. You might grab a large block of memory that needs to be freed up afterwards.

It isn't just about resources. Parts of our program might need to set a particular processing context - for example it might need to change the default mathematical precision, or catch runtime warnings. These states need to be undone when you are finished, otherwise the rest of the code might not function exactly as expected.

It is possible to undo these states manually, but there are dangers to this. If the code throws an exception, the code block might exit without cleaning up properly. If the code is very complex, with multiple exit points, you might miss a case, and exit without restoring the initial state.

With statements provide a neat way of handing this. When you execute a with statement, it creates a context manager that creates the new context. When your code exits the body of the with statement, for any reason, the context manager is called again to clean up.

In this article we will look at the most common use, ensuring that files are closed correctly after use.

Manually closing files

The most commonly used system resources are probably file handles. Every time you open a file in Python, it uses grabs file handle to tell the system that it is using the file. When Python closes the file, it gives that handle back. Here is some typical code that does this:

f = open('text.txt', 'r') # Grabs a file handle
s = f.read()
f.close()                 # Gives the handle back

File handles are global resources, shared by every program running on the system, and there are only a finite number of them. Modern computers have quite a lot of file handles, but even so it is good practice to always close a file when you have finished with it.

In addition, if you fail to close a file when you have finished with it, it can cause problems if other programs need to access the same file.

Manually closing the file, similar to the code above, works most of the time. But there are cases when it can fail:

  • If the code throws an exception (for example, if the file read operation failed), the code would jump straight to the exception handler and close would never be called. Note that this would happen with any kind of exception, not just a file exception. If there was extra code in that block that threw a divide by zero exception, it could still prevent close being called, which could cause problems.
  • In more complex code, there might be more than one way to exit the code block - for example there might be return statements or break statements. It can be difficult to be certain that every possible case has been handled, and the code will always call close.

This is where a with statement is useful.

Using with statements

In the example above, we could use a with statement, like this:

with open('text.txt', 'r') as f:
    s = f.read()

The with statement calls open and assigns the returned file object to the variable f - just like our previous open call in the original code.

When Python exits the with block for any reason the file f will get closed. this happens if there is an exception, a return, a break, or if the code block just ends normally. In every case, close will be called.

Notice that we don't even need to call close explicitly in our code (we could, but it would be pointless). We just let the with block do its thing.

We can nest with blocks, for example to do a file copy:

with open('text.txt', 'r') as fin:
    with open('out.txt', 'w') as fout:
        s = fin.read()

When the inner block ends, both the files will be closed. In addition, if the output file failed to open, the input file would still be closed when its with block terminated.

An alternative way to create a nested with loop is this:

with open('text.txt', 'r') as fin, open('out.txt', 'w') as fout:
        s = fin.read()

Which form you choose is a matter of personal choice, although of course if it results in excessive line lengths that should usually be avoided.

The context manager

A with statement operates with a context manager:

with contextmanager as x:

A context manager is an object that implements __enter__ and __exit__ methods. The with statement calls the enter method as it enters the with code block, then calls the exit method when control leaves the with code block for any reason.

The file example works because a Python File object is a context manager. In particular, it has and exit method that closes the file.

A later article will cover other scenarios.

See also

If you found this article useful, you might be interested in the book NumPy Recipes or other books by the same author.

Join the PythonInformer Newsletter

Sign up using this form to receive an email when new content is added:

Popular tags

2d arrays abstract data type alignment and angle animation arc array arrays bar chart bar style behavioural pattern bezier curve built-in function callable object chain circle classes clipping close closure cmyk colour combinations comparison operator comprehension context context manager conversion count creational pattern data science data types decorator design pattern device space dictionary drawing duck typing efficiency ellipse else encryption enumerate fill filter font font style for loop formula function function composition function plot functools game development generativepy tutorial generator geometry gif global variable gradient greyscale higher order function hsl html image image processing imagesurface immutable object in operator index inner function input installing iter iterable iterator itertools join l system lambda function latex len lerp line line plot line style linear gradient linspace list list comprehension logical operator lru_cache magic method mandelbrot mandelbrot set map marker style matplotlib monad mutability named parameter numeric python numpy object open operator optimisation optional parameter or pandas partial application path pattern permutations pie chart pil pillow polygon pong positional parameter print product programming paradigms programming techniques pure function python standard library radial gradient range recipes rectangle recursion reduce regular polygon repeat rgb rotation roundrect scaling scatter plot scipy sector segment sequence setup shape singleton slice slicing sound spirograph sprite square str stream string stroke structural pattern subpath symmetric encryption template tex text text metrics tinkerbell fractal transform translation transparency triangle truthy value tuple turtle unpacking user space vectorisation webserver website while loop zip zip_longest