Managing Contexts: the with Statement

Many objects manage resources, and must impose a rigid protocol on use of that resource.

For example, a file object in Python acquires and releases OS files, which may be associated with devices or network interfaces. We looked at the way that file objects can be managed by the with statement in File Statements.

In this section, we’ll look at ways in which the new with statement will simplify file or database processing. We will look at the kinds of object design considerations which are required to create your own objects that work well with the with statement.

Important

Legacy

In versions of Python prior to 2.6, we must enable the with statement by using the following statement.

from __future__ import with_statement

Semantics of a Context

While most use of the with statement involve acquiring and releasing specific resources – like OS files – the statement can be applied somewhat more generally. To make the statement more widely applicable, Python works with a context. A context is not limited to acquiring and releasing a file or database connection. A context could be a web transaction, a user’s logged-in session, a particular transaction or any other stateful condition.

Generally, a context is a state which must endure for one or more statements, has a specific method for entering the state and has a specific method for exiting the state. Further, a context’s exit must be done with the defined method irrespective of any exceptions that might occur within the context.

Database operations often center on transactions which must either be completed (to move the database to a new, iternally consistent state,) or rolled back to reset the database to a prior consistent state. In this case, exceptions must be tolerated so that the database server can be instructed to commit the transaction or roll it back.

We’ll also use a context to be sure that a file is closed, or a lock is released. We can also use a context to be sure that the user interface is reset properly when a user switches their focus or an error occurs in a complex interaction.

The design pattern has two elements: a Context Manager and a Working Object. The Context Manager is used by the with statement to enter and exit the context. One thing that can happen when entering a context is that a Working Object is created as part of the entry process. The Working Object is often used for files and databases where we interact with the context. The Working Object isn’t always necessary; for example acquiring and releasing locks is done entirely by the Context Manager.

Using a Context

There are a few Python library classes which provide context information that is used by the with statement. The most commonly-used class is the file class.

There are two forms of the with statement. In the first, the context object does not provide a context-specific object to work with. In the second, the context provides us an object to be used within the context.

with context :
    suite
with context as variable :
    suite

We’ll look at the second form, since that is how the file class works. A file object is a kind of context manager, and responds to the protocol defined by the with statement.

When we open a file for processing, we are creating a context. When we leave that context, we want to be sure that the file is properly closed. Here’s the standard example of how this is used.

with file('someData.txt','r') as theFile:
   for aLine in theFile:
       print aLine
# the file was closed by the context manager
  1. We create the file, which can be used as a context manager. The with statement enters the context, which returns a file object that we can use for input and output purposes. The as clause specifies that the working object is assigned to theFile.
  2. This is a pretty typical for statement that reads each line of a file.
  3. The with statement also exits the context, irrespective of the presence or absence of exceptions. In the case of a file context manager, this will close the file.

In the previous example, we saw that the file factory function is used to create a context manager. This is possible because a file has several interfaces: it is a context manager as well as being a working file object. This is potentially confusing because it conflate file context manager with the working file object. However, it also

This has the advantage of making the with statement optional. In some simple applications, improperly closed files have few real consequences, and the carefully managed context of a with statement isn’t necessary.

Defining a Context Manager Function

One easy way to create a context manager is to write a function that handles the acquisition and release of an important resource.

The contextlib module provides a handy decorator that transforms a simple generator function into a context manager.

Conceptually, you’ll break your function into two phases:

  • The __enter__() phase acquires the resources. This is written as statements from the start of the function’s suite to up to the one and only yield statement. The value that is yielded is the value that’s used in the as clause.

    Once the function yields this value, then the with statement’s suite starts processing.

    If an exception occurs anywhere in the with statement, it will be raised by the yield statement, and must be handled by the function to assure that resources are released properly.

  • The __exit__() phase releases resources. This is written as statements after the yield statement.

Here’s an example of a context manager that manages a subclass of file. The FileWithCount has an extra method that will append a summary line that shows if everything has gone properly.

We’ll manage this object so that we are assured that file either has a summary, or is removed from the directory.

import os

class FileWithCount( file ):
    def __init__( self, *args, **kw ):
        super( FileWithCount, self ).__init__( *args, **kw )
        self.count= 0
    def write( self, data ):
        super( FileWithCount, self ).write( data )
        self.count += data.count('\n' )
    def writelines( self, dataList ):
        for d in dataList:
            self.write( d )
    def summarize( self ):
        super( FileWithCount, self ).write( "\n:count:`%d`\n" % self.count )
  1. We defined a FileWithCount class that adds a line-counting feature to the built-in file type.
  2. Note that we’re defining a subclass of file that adds features. For the most part, we simply pass the arguments to the superclass method functions.
  3. The write() method counts the number of newline characters written to the file.
  4. The summarize() method appends a label with final count to the end of the file.

Here’s a context manager that uses our FileWithCount class.

from contextlib import contextmanager

@contextmanager
def file_with_count( *args, **kw ):
    # The __enter__ processing
    try:
        counted_file=  FileWithCount( *args, **kw )
        yield counted_file
        # The __exit__ processing -- if everything's ok
        counted_file.summarize()
        counted_file.close()
    except:
        # The __exit__ processing -- if there as an exception
        counted_file.close()
        os.remove( counted_file.name )
        raise
  1. We defined a context manager function, file_with_count() that builds an instance of FileWithCount and yields it to a with statement.
  2. If everything works normally, the counted_file.summarize() statement is executed.
  3. If there is an exception, then the counted_file.close() statement is executed, and the file is removed via os.remove(). The is removed so that incomplete files are not left around to confuse other application programs or users.

Here’s an example of using this context manager to create a temporary directory.

with file_with_count( "file.data", "w" ) as results:
    results.write( "this\nthat\n" )

This yields a file with the following content.

MacBook-5:Python-2.6 slott$ cat file.data
this
that

:count:`2`

The final :count: line is written automatically by the context manager, simplifying our application.

Defining a Context Manager Class

In some cases, a class is both the working object and the context manager. The file class is the central example of this. The two do not have to be tied together. It’s clearer if we’ll look at creating a context manager that is separate from the working object first.

A Context Manager must implement two methods to collaborate properly with the with statement.

ContextManager.__enter__(self) → object
This method is called on entry to the with statement. The value returned by this method function will be the value assigned to the as variable.
ContextManager.__exit__(self, exc_type, exc_val, exc_tb) → boolean

This method is called on exit from the with statement. If the exc_type, exc_val or exc_tb parameters have values other than None, then an exception occured.

  • A return value of False will propagate the exception after __exit__() finishes.
  • A return value of True will suppress any exception and finish normally.

If the exc_type, exc_val or exc_tb parameters have values of None, then this is a normal conclusion of the with statement. The method should return True.

Here is a context manager class, named FileCountManager, which incorporates the FileWithCount class, shown above. To be a context manager, this class implements the required __enter__() and __exit__() methods.

class FileCountManager( object ):
    def __init__( self, *args, **kw ):
        self.theFile= FileWithCount( *args, **kw )
    def __enter__( self ):
        return self.theFile
    def __exit__( self, exc_type, exc_val, exc_tb ):
        if  exc_type is not None:
            # Exception occurred
            self.theFile.close()
            os.remove( self.theFile.name )
            return False # Will raise the exception
        self.theFile.summarize()
        self.theFile.close()
        return True # Everything's okay
  1. The __enter__() method creates the FileWithCount and returns it so that the as clause will assign this to a variable.

  2. The __exit__() method checks to see if it is ending with an exception.

    In case of an exception, we close the file and remove it. We also return False to permit the exception to propagate outside the with statement.

    If there was no exception, then the :class:FileWithCount.summarize` is used to write the summary and the file is closed.

The overall main program can have the following structure. We don’t need to make special arrangements in the main program to be sure that the log is finalized correctly. We have delegated those special arrangements to the context manager object, leaving us with an uncluttered main program.

with FileCountManager( "file.data", "w" ) as results:
    results.write( "this\nthat\n" )

Context Manager Exercises

  1. Build a class with it’s own manager. Merge the methods from FileCountManager into FileWithCount to create a single class which does both sets of features.

  2. List with a Checksum. A crypto application works with lists, but only lists that have a checksum of the values in a list of numbers.

    Define a class, ManageChecksum, which removes and replaces the last element in a non-empty list.

    • The __init__() method accepts a single parameter which is a list or MutableSequence object.
    • On __entry__(), given a zero-length list, return the list object.
    • On __entry__(), given a list with 1 or more elements, pop the last element. Assert that this is the cryptological checksum of the values of the other elements. Return this updated list.
    • On __exit__(), comput cryptological checksum of the elements and append this checksum to the list.

    For now, our cryptological checksum can be the simple sum, created with sum(). As an advanced exercise, look at using hashlib to put a better checksum on the list.

    You should be able to do the following kinds of test.

    crypto_list = []
    with ManageChecksum( crypto_list ) as theList:
        theList.extend( [5, 7, 11] )
        assert theList == [5, 7, 11] # no checksum in the with statement
        theList.append( 13 )
    assert theList[:-1] == [5, 7, 11, 13] # checksum was added to prevent tampering
    with ManageChecksum( crypto_list) as theList:
        theList.pop( 0 )
    assert theList[:-1] == [7, 11, 13]
    

    Inside the with statement, the list is an ordinary list.

    Outside the with statement, the list has an anti-tampering checksum appended.

Table Of Contents

Previous topic

Decorators

Next topic

Modules

This Page