Building Skills in Programming

How To Write Your Own Software Using Python


The Walrus and the CarpenterLewis Carroll

“The time has come,” the Walrus said,
“To talk of many things:
Of shoes – and ships – and sealing-wax –
Of cabbages – and kings –
And why the sea is boiling hot –
and whether pigs have wings.”

Getting Started

Tools and Toys

This part provides some necessary background to help non-programming newbies get ready to write their own programs. If you have good computer skills, this section may be all review. If you are very new to computers, our objective is to build up your skills by providing as complete an introduction as we can. Computing has a lot of obscure words, and we’ll need some consistent definitions.

We’ll start with the big picture. In About Computers we’ll provide a list of concepts that are central to computers, programs and programming. In About Programs we’ll narrow our focus to programs and how we create them.

In Let There Be Python: Downloading and Installing we’ll describe how to install Python. You’ll need to choose just one of Windows Installation, Macintosh Installation or GNU/Linux and UNIX Overview. This chapter has the essential first step in starting to build programs: getting our tools organized.

We’ll describe two typical problems that Python can help us solve in Two Minimally-Geeky Problems : Examples of Things Best Done by Customized Software. We’ll provide many, many more exercises and problems than just these two. But these are representative of the problems we’ll tackle.

We also provide some history and background to help show why Python is so cool. If you are already convinced that Python is your tool of choice, you can skip Why Python is So Cool. If you’ve heard about Visual Basic, Java or C++ and wonder why Python is better, you might find something helpful in that section. It involves some computer-science jargon; you’ve been warned.

Using Python

Taking Your First Steps

Now that you have Python installed, we can start using it. We’ll look at a number of ways that we can interact with the Python application. We’ll use these interactions to learn the language.

In later sections, after we’ve got a more complete grip on the language and start to write programs, we’ll move on to more advanced ways to use the Python program. Our goal is to use Python in an automated fashion to do data processing. Here in phase one, we’ll be using Python manually to learn the language.

We’ll describe the direct use of the python to process Python-language statements in Instant Gratification : The Simplest Possible Conversation. This will help us get started; it provides immediate gratification, but isn’t the easiest way to work.

We’ll dig into IDLE in IDLE Time : Using Tools To Be More Productive. We’ll emphasize this as a good way to learn the language as well as build programs.

Arithmetic and Expressions

Before Reading and ‘Riting comes ‘Rithmetic

The heart of Python is a rich variety of numeric types and arithmetic operators. We can use these various numeric types to do basic mathematical operations on whole numbers, real numbers and complex numbers. We’ll look at the basics in Simple Arithmetic : Numbers and Operators.

In addition to the basic arithmetic capabilities, many kinds of problems need additional mathematical and financial functions. We’ll look at some of the built-in functions and some functions in add-on modules in Better Arithmetic Through Functions and Extra Functions: math and random.

For more specialized problems, Python has a variety of additional operators. We’ll look more deeply at these additional operators in Special Ops : Binary Data and Operators.

We’ll cover some optional topics in Peeking Under the Hood, including different approaches to execution of Python statements, some notes on Python writing style.

Programming Essentials

The Input-Process-Output Pattern

We often define programs using the pattern “input-process-output” . We’ll work through this sequence backwards. In order to see output from a script, we’ll need to use the print statement. We’ll look at this in Seeing Results : The print Statement.

Once we are comfortable with the print statement, we can introduce processing in Turning Python Loose With a Script. When we start making more finished and polished programs, we’re going to want to make them easy to use. There are a lot of options and shortcuts available to us, and we’ll touch on a few of them here. Later, we’ll add even more ease-of-use features.

We In order to do processing, we’ll introduce variables and the assignment statement in Expressions, Constants and Variables. This will allow us to do the basic steps of processing. We’ll describe some additional features in Assignment Bonus Features

When we add input in Can We Get Your Input?, we’ll have all three parts to the input-process-output pattern.

Some Self-Control

Making Choices, Doing It All

This section represents a significant milestone. Up until this part, we have presented Python as a souped-up desk calculator. This part shows the essential elements of building automated data processing.

We’ll start with truth and logic in Truth and Logic : Boolean Data and Operators. This is an extension to the expressions and numeric types we started out with. We’ll add another data type, boolean , and a number of operators, including comparisons and basic logic of and, or and not. With this foundation in logic, we can introduce comparisons in Making Decisions : The Comparison Operators.

The basic tools of logic and comparison are the essential ingredient to looking at conditional processing in Processing Only When Necessary : The if Statement. Conditional processing is controlled by the if statement, and reflects processing that only makes sense when a condition is true.

The other side of this is iterative processing, which we’ll cover in While We Have More To Do : The for and while Statements. Iterative processing also depends on a condition, but it iterates (or repeats) while the condition is true. Python provides two statements for this, the for statement and the while statement.

We’ll cover a number of additional topics Becoming More Controlling. This includes the break , continue and assert statements to provide a finer level of control over the processing. Additionally, we’ll look at many of the traps and pitfalls associated with iterative processing.

In Turning Python Loose with More Sophisticated Scripts we’ll return to scripting to make our scripts fit more smoothly with our operating system. We’ll look at a complete family tree of processing alternatives using the command line as well as the GUI. There are a number of operating-system specific variations on this theme, and we can’t easily cover every alternative.

Organizing Programs with Function Definitions

Building A Solution in Pieces

Our initial programs have been sequences of statements. As our programs get more complex, we will find that this style of “long, flat” program is hard to work with. In Adding New Verbs : The def Statement we’ll introduce the primary method for structuring and organizing our application programs, the function. It turns out that breaking a program into separate functions allows us to decompose a solution into several simpler parts. Functions are also a good intellectual tool to help us divide and conquer a complex problem.

We’ll add several useful features in Flexibility and Clarity : Optional Parameters, Keyword Arguments. These will add flexibility so that it’s easier to understand and use the functions we define.

In A Few More Function Definition Tools, we’ll show a number of unique features that make Python’s function definitions much cooler than other programming languages

Getting Our Bearings

In sailing terms, we’re rounding the mark : we’ve finished one leg of our journey and we’re starting the next leg. Many sailing race courses are variations on a simple out and back design. When racing on one of these courses, you cover the same water going in opposite directions. Many courses are laid out so that you start the race going into the wind and finish the race going away from the wind.

Following this pattern, we’re covering programming by first viewing it as procedural statements and then viewing it as data structures. Neither, by itself, is a complete picture. Each depends on the other: we need data to process with our procedural statements; we need procedural statements to process our data.

Data and processing are two sides to the same coin. This duality is central to all programming, and leads to a terrible dilemma when teaching programming: which comes first? We can’t easily teach data without the statements to process it. Neither can we teach the processing statements without covering the associated data structures.

Basic Sequential Collections of Data

Strings, Lists and Tuples

Python has a rich family tree of collections. This part will focus on the sequential collections; Collecting Items in Sequence will introduce the features that are common to all of the types of sequences.

In Sequences of Characters : str and Unicode we describe the string subclass of sequence. The exercises include some challenging string manipulations.

We describe fixed-length sequences, called tuples in Doubles, Triples, Quadruples : The tuple. Because tuples are quite simple, they give us an opportunity to digress and introduce some basic kinds of algorithms commonly used for statistical processing. The exercises include Translating From Math To Python: Conjugating The Verb “To Sigma”, which describes how to approach writing programs for doing statistical calculations.

In Flexible Sequences : the list we describe the variable-length sequence, called a list. Lists are one of the cool features that set Python apart from other programming languages. The exercises at the end of the list section include both simple and relatively sophisticated problems.

We’ll cover some advanced features of the list in Common List Design Patterns. This chapter includes some common techniques for creating useful data structures out of the basic tools we have at our disposal. It will cover the common need to sort a list into order. We’ll also cover multi-dimensional structures: moving from mathematical vectors to matrices.

Additional Processing Control Patterns

Exceptions and Iterators

Exception processing is a way to alter the ordinary sequential execution of the statements in our program. Additionally, an Exception is an object that is raised internally by Python when our program does something illegal. We can make considerable use of exceptions and exception-handling statements to create event-driven programs. We’ll cover this in The Unexpected : The try and except statements.

In Looping Back : Iterators, the for statement, and the yield statement we’ll look closely at some advanced procedural processing. We’ll look at a Python object called an iterator and how we can create generator functions. These will allow us to define some more sophisticated processing; processing that will help us cope with the kinds of files we often encounter in the real world.

Working with Files

The Permanent Record

Files are one of the most important features of our operating system. Back in Hardware Terminology we talked about various kinds of hardware on which files reside.

In Software Terminology we talked briefly about the operating system structure and protocol that defines the “files” or “documents” we work with. The operating system insulates us from the complexities of various devices and provides us some handy abstractions that make it much easier to save, find and manage our documents.

Up until now, we’ve used very few files in just three ways. We’ve used a file named python (Windows python.exe) heavily. This file contains the binary program that is Python; we’ve run this program by typing a command in a terminal window or double-clicking an icon. We’ve also used a file named; this is a Python script that contains the IDLE program. Finally, we’ve also saved our various scripts in files and asked Python to run those files.

One of the most important things that our programs can do is read or write files. Files are a mixture of two unrelated concepts: they are a collection of data items, and they involve our OS notion of a file system, file names, directories, and devices. We’ll introduce files in External Data and Files. We’ll add the OS processing in Files II : Some Examples and Some Modules. We’ll wrap up with an overview of all the things files are used for in Files III : The Grand Unification.

Data + Processing = Objects

One of the most powerful and useful features of Python is its ability to define new classes of data. The next chapters will introduce the class definition and the basics of object-oriented programming.

Objects: A Retrospective reviews the key features of the objects we’ve used so far. Having looked at what we’ve already learned, we can then introduce the basics of how we define the class of an object in Defining New Objects. Inheritance, Generalization and Specialization introduces simple inheritance. We extend this discussion further to include several common design patterns that use polymorphism. Additional Classy Topics adds some additional functions and programming techniques.

Special Behavior Requires Special Methods introduces the special methods. New Kinds of Numbers: Fractions and Currency describes the mechanism for adding types to Python that behave like the built-in numeric types. We can also add new collection types; the optional material in Creating New Types of Collections may help you understand more of how Python works.

Modules : The unit of software packaging and assembly

The basic Python language is rich with built-in features. These include several sophisticated built-in data types, numerous basic statements, a variety of common arithmetic operators and a library of built-in functions. As cool as Python is, the real power lies outside this kernel of built-in features. Python’s real strength lies in the vast number of extension modules that add specialized features for problems of every kind.

There are a vast (and growing) set of powerful and sophisticated features of Python in the library of extension modules. There are several advantages to this arrangement. First, it allows modules to be added easily, further extending the power of Python. Second, it allows each program to load only the relevant modules, keeping your program as simple as possible. Third, it allows a module to be replaced, allowing the developer to choose among competing components to create the best solution to a problem.


The Batteries Included Principle

In the Python community, the ease with which people add modules to Python leads to the Batteries Included principle. Many people use Python because there is almost always some group of modules which are directly applicable to their problems.

Making Python appropriate to a wide variety of problems absolutely requires that everyone with a good idea can add a new module to Python. You know it’s a good idea when you’ve used the same module a few times, and other people have asked you for it.

We’ve already made use of several modules. In The math Module – Trig and Logs we covered math and random modules. In Files II : Some Examples and Some Modules we touched on several modules: sys, glob, fnmatch, fileinput, os, os.path, tempfile, and shutil.

We’ll do the formal introductions to modules in Module Definitions – Adding New Concepts.

We’ll include general overview of the Python modules in Essential Modules : The Python Library. We’ll look at the decimal module in Fixed-Point Numbers : Doing High Finance with decimal. Time and Date Processing : The time and datetime Modules covers time , and datetime . We’ll also look at more sophisticated the re module in Text Processing and Pattern Matching : The re Module.

Fit and Finish: Complete Programs

We’ve covered almost all of the statements of the Python language, and all of the useful built-in types. We’ve seen how to organize our programs using functions, classes and modules. We’ve seen how to define our own types. At this point, we have almost everything we need to write useful programs.

This part covers the remaining topics in producing a polished result. We’ll look at the basic principles of an application script in Wrapping and Packaging Our Solution. We’ll look at four different common patterns for finished programs in Architectural Patterns – A Family Tree. Finally, in Professionalism : Additional Tips and Hints, we’ll look at some additional techniques that make your project into a professional, polished product.

Appendix 1 : Debugging

This appendix lists all of the debugging tips in one handy reference section.

Indices and Tables

Other Back Matter

The following toolset was used for production of this book.

  • Python 2.6.3.
  • Sphinx 0.63.
  • Docutils 0.4.
  • Komodo Edit 5.2.
  • pyPDF 1.12.
  • MacTeX-2008.