James Gardner: Home > Blog > 2009 > Writing a Python Command Line...

Writing a Python Command Line Interface (cli) with Sub-Commands

Posted:2009-04-25 19:39
Tags:Python

Update 2009-07-30: I've now officially released CommandTool here.

Contents

I've been writing a tool for managing the static files in a website based on Dreamweaver-style templates and I wanted a command line interface akin to the the likes of subversion or mercurial where the main command has a series of sub commands eg:

The terminology I'm using is that the main command is called the program and the sub commands are called commands. The options directly associated with the program are called program options and the options associated with the command are called command option.

In the example above, svn is the program, st and ci are aliases for the commands status and commit respectively, --help is a program option and -m is a command option.

My tool is called due (which happens to stand for Deliberately Under-Engineered, more on that in another post) and I want the Python API to exaclty match the command line interface. This means that:

To make matters slightly more complicated there is also a config file which allows you to set default for variables which are overridden on the command line. These variables are known as metavars in my terminology. There is also a plugin architecture so that different plugins can be responsible for different parts of the site. For example the BlogPlugin class handles all pages and index lists in the /blog directory. Each plugin needs an oppurtunity to parse the config file to specify default values for its own metavars.

My first implementation used the Python optparse module. I created a class called SwitchCommand which took a series of OptionParser classes and switched between them based on the second part of the command so that due create would call the cmd_create OptionParser instance whereas due convert would call cmd_convert. This seemed like it would work well to start with but the more I used it the more I got dissatisfied with the decisions the optparse authors had made, they just didn't fit my use case of a main program and series of commands. I ended up introducing more and more hacks to unpick the way optparse works.

Note

The optparse module is brilliant if you are just handling a simple program so don't be put off. It is just that if you are trying to produce something that it wasn't designed for it will be hard work.

My Requirements

I need the user to be able to enter a set of commands with this sort of structure:

due [PROGRAM_OPTIONS] COMMAND [COMMAND_OPTIONS] ARGS

I want to also allow users to put the options and arguments in any order with the one proviso that if both the program itself and the command which will be run take the same option, for example --help, if the option comes before the command name it will be treated as a program option, otherwise it will be treated as a command option.

Conceptulising the Problem

This means there a 3 sets of options which each need to be handled differently by the command line parsing tool:

PROGRAM_OPTIONS

These are options which are not used by any of the commands and which can therefore appear anywhere on the command line without causing confusion. For example, these might all be treated as program options:

Program Options:
  --version             show program's version number and exit
  -q, --quiet           don't print status messages to stdout
  -v, --verbose         print debug messages as well as usual output

SHARED_OPTIONS

These are options which are used by both the program and the command and which result in different actions depending on whether they appear before or after the command name. For example, consider the shared option --help:

Shared Options:
  --help                show this help message and exit

As an example:

$ due --help

would list all program commands whereas:

$ due create --help

would display the help for the create sub-command.

Shared options must therefore be placed in the correct place.

COMMAND_OPTIONS

These are options which only affect the command in question. Since they aren't shared options they can appear anywhere.

API Considerations

With this in mind let's think about how the API might work.

Aliases

I want the commands to be able to be referenced by aliases so that a user can run due rc rather than due recreate and have the same action performed.

Return Value

I want the parser to return four data structures:

program_options
A dictionary containing the PROGRAM_OPTIONS and any SHARED_OPTIONS associated with the program rather than the command being used (if any).
command_options
A dictionary of all the COMMAND_OPTIONS and any SHARED_OPTIONS associated with the command being run rather than program (if any).
command
The name of the command itself (not its alias)
args
A list of any extra arguments specified (excluding the COMMAND itself)

Specifying Available Options

In order for the command line parser to spot errors it needs to know all the program options and all the command options for the program and every possible command. I specifically don't want different commands to use the same option to mean different things as this could make the commands seem inconsistent to the user so having them all defined to start with is fine.

Note

If you are building a command line interface where different commands use the same option to use different things, the approach I'm taking here won't work well for you.

Internal Variables

Conceptually it is helpful to think about the set of command line options affecting an internal variable in the application. For example, both the --quiet and --verbose options can be thought of as affecting an internal variable called verbosity.

After the command line is parsed I'd like to be able to access the options by the internal variable they affect so that I can easily work out the correct value the verbosity variable has internally based on the options specified.

Metavars

Some options are just flags which don't have a value associated with them, for example --help, --quiet etc. Others do have a value and this can be given a label which I'm calling a metavar. For example --database=users. If we defined that the --database option set the metavar DATABASE, the DATABASE metavar would then contain the value users if this option was used.

Each internal variable can have more than one metavar associated with it. For example consider the case where you have an internal variable postition representing the lattitude and longitude coordinates where a photo was taken. The correspondind opitons to set these variables might be --latitude=LATITUDE, --longitude=LONGITUDE``, thus the position internal variable has two metavars associated with it, LATITUDE and LONGITUDE.

The same metavar can also affect more than one internal variable. For example if you had a metavar called BASE_PATH in a move command it might affect both the source and dest internal variables. It also means that options can be re-used amongst internal variables.

The only restriction is that the same metavar can't take two different values in the same command so you have to take care to ensure that metavars are given different names if they are allowed to appear in the same command with different values. Such a condition raises an Exception, instead of printing an error.

Caution!

Because the same config option can be used with multiple internal variables, it means that in the program_options and command_options dictionaries, the one option might result in multiple values being present, one for each internal variable it affects. This is the designed behaviour but it might catch you out if aren't expecting duplicated options in the results.

Config Files

Now that you have the concept of metvars associated with options affecting internal variables, it is easy to intoriduce the concept of a config file.

A config file is simply a format which, when parsed, results in a Python dictionary where the keys are the metavars and the values are the values associated with a metavar. For example, a config file might look like this:

LATITUDE  66° 33' 39'' N

On the command line the --latitude option would no longer need to be set because the value from the LATITUDE metavar in the config file could be used instead.

Note

Notice that the config file doesn't specify the internal variable, just the metavar. This means the application code which interprets the command line options can behave in exactly the same way regardless of whether the metavars were specified on the command line or from a config file.

There is currently no way to specify flags like --help or --quiet in a config file and no way to specify arguments. I might implement such features if I come accross a suitable use case that can't be handled with the existing functionality.

Niceties

Almost all command line programs share some common features which can be automated to some degree. All this functionality is optional though and can be ignored in your own comand line program if you prefer.

Error and Try Message

If you enter a set of invalid options or arguments or a command which is not regonised, an error is displayed but is also helpful to print some information explaining how the user can get help to find out what the correct flag are. For example, look at how the mv command works on Linux:

$ mv /home/james/example
mv: missing destination file operand after `/home/james/example'
Try `mv --help' for more information.

The error message will be displayed by the parsing code but the try message can be configured.

Some command line programs print the usage and help when an error occurs but I value screen space and don't like that approach. Instead my applications will display the help if the --help flag is displayed anywhere on the command line so that you can just press the up arrow type --help and press enter to get the help without having to write a new command from scratch.

Help Text

The help text will take a different format depending on whether it is help with the program overall or with a specific command.

Help messages usually have the following components:

  • usage - explains how to structure the options and what argument are expected
  • description - explains what the command does and what the arguments are for
  • options - explains what each of the options do

Program help will usually also have a section listing the available commands. For example, here's the svn program's help output:

$ svn --help
usage: svn <subcommand> [options] [args]
Subversion command-line client, version 1.5.1.
Type 'svn help <subcommand>' for help on a specific subcommand.
Type 'svn --version' to see the program version and RA modules
  or 'svn --version --quiet' to see just the version number.

Most subcommands take file and/or directory arguments, recursing
on the directories.  If no arguments are supplied to such a
command, it recurses on the current directory (inclusive) by default.

Available subcommands:
   add
   blame (praise, annotate, ann)
   cat
   changelist (cl)
   checkout (co)
   cleanup
   commit (ci)
   copy (cp)
   delete (del, remove, rm)
   diff (di)
   export
   help (?, h)
   import
   info
   list (ls)
   lock
   log
   merge
   mergeinfo
   mkdir
   move (mv, rename, ren)
   propdel (pdel, pd)
   propedit (pedit, pe)
   propget (pget, pg)
   proplist (plist, pl)
   propset (pset, ps)
   resolve
   resolved
   revert
   status (stat, st)
   switch (sw)
   unlock
   update (up)

Subversion is a tool for version control.
For additional information, see http://subversion.tigris.org/

The individual command's help will take a similar format but without the list of available sub-commands. For example, here's the help output from svn ci --help:

$ svn ci --help
commit (ci): Send changes from your working copy to the repository.
usage: commit [PATH...]

  A log message must be provided, but it can be empty.  If it is not
  given by a --message or --file option, an editor will be started.
  If any targets are (or contain) locked items, those will be
  unlocked after a successful commit.

Valid options:
  -q [--quiet]             : print nothing, or only summary information
  -N [--non-recursive]     : obsolete; try --depth=files or --depth=immediates
  --depth ARG              : limit operation by depth ARG ('empty', 'files',
                            'immediates', or 'infinity')
  --targets ARG            : pass contents of file ARG as additional args
  --no-unlock              : don't unlock the targets
  -m [--message] ARG       : specify log message ARG
  -F [--file] ARG          : read log message from file ARG
  --force-log              : force validity of log message source
  --editor-cmd ARG         : use ARG as external editor
  --encoding ARG           : treat value as being in charset encoding ARG
  --with-revprop ARG       : set revision property ARG in new revision
                             using the name[=value] format
  --changelist ARG         : operate only on members of changelist ARG
                             [aliases: --cl]
  --keep-changelists       : don't delete changelists after commit

Global options:
  --username ARG           : specify a username ARG
  --password ARG           : specify a password ARG
  --no-auth-cache          : do not cache authentication tokens
  --non-interactive        : do no interactive prompting
  --config-dir ARG         : read user configuration files from directory ARG

The optparse module generates these help messages for you with a small degree of flexibility to customise how they appear. There are two problems with this for more complex applications:

  • You might want extra information displayed or help formatted in a different way
  • You might not want to expose every possible option in the help text, instead encouraging the user to use the most up-to-date API

It is easy to generate the help messages yourself as static text anyway so I'd recommend doing that rather than trying to auto-generate help.

In order to make this process easier though, the docstring of the function which handles the specific command can be used as the help text, that way you only need to maintain the help text in one place.

Introducing commandtool

Having written all the above requirements I thought it made sense to write a tool which implements them. You can download commandtool here.

The rest of this article explains how to use it to build command line interfaces with the features described so far.

We'll describe an example program for finding words or files called example which has two sub-commands, name and content. The first is used to search for filenames, the second for words inside files. The name command will have two aliases: n and nm and the content command will have aliases c and ct. Both commands will only look in one directory.

Python API

The Python API looks like this (I know it isn't great functionality, this is about the command line API, not the implementation of a search!):

import os.path
import logging
log = logging.getLogger('find')

def name(letters, start_directory='/home/james'):
    found = []
    log.info(
        'Finding files in %r whose filenames contain the letters %r',
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)):
            log.debug('Trying %r', filename)
            if letters in filename:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

def content(letters, start_directory='/home/james', file_type='.txt'):
    found = []
    log.info(
        'Finding %r files in %r which contain the letters %r',
        file_type,
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)) and filename.endswith(file_type):
            log.debug('Trying %r', filename)
            fp = open(os.path.join(start_directory, filename), 'r')
            data = fp.read()
            fp.close()
            if letters in data:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

You can see that there are three internal variables being used: start_directory, letters and file_type. In both cases the variable letters is required. It should therefore be implemented as an argument rather than an option. start_directory and file_type can be options.

Option Sets

We will also want an internal variable called verbose which will be configured by program options and affect the log level which gets printed to the console. We'll also want a help variable which should affect whether the help text for the program or the command should be shown.

Let's look at the options definition we'll need:

option_sets = {
    'start_directory': [
        dict(
            type = 'command',
            long = ['--start-directory'],
            short = ['-s'],
            metavar = 'START_DIRECTORY',
        ),
    ],
    'file_type': [
        dict(
            type = 'command',
            long = ['--file-type'],
            short = ['-t'],
            metavar = 'FILE_TYPE',
        ),
    ],
    'help': [
        dict(
            type = 'shared',
            long = ['--help'],
            short = ['-h'],
        ),
    ],
    'config': [
        dict(
            type = 'command',
            long = ['--config'],
            short = ['-c'],
            metavar = 'CONFIG',
        ),
    ],
    'verbose': [
        dict(
            type = 'program',
            long = ['--verbose'],
            short = ['-v'],
        ),
        dict(
            type = 'program',
            long = ['--quiet'],
            short = ['-q'],
        ),
    ]
}

Notice that:

  • the options are grouped by the internal variable name they affect
  • you can specify both long and short option names for the same internal variable
  • you can specify more than one set of options for the same internal variable if you prefer
  • each option set a type specifying whether the option can only be used in programs ('programs'), only be used in command ('command') or can be used in either ('shared')
  • options sets where the options set a value have a metavar assoicated with them

There are some other things you should be aware of:

  • you can't use the same option in two option sets
  • although you can specify more than one long and short option, only the first of each appears in any help output by default

Aliases

The aliases are easy to set up, they look like this:

aliases = {
    'name': ['n', 'nm'],
    'content': ['c', 'ct'],
}

Metavar Handlers

Sometimes it is useful to have some pre-processing performed on option values (metavars). For example you might like to convert all path-related metavars from the value specified on the command line into absolute, normalized paths before handling them in the application. Handlers can perform this task for you.

Handlers are simply functions which take metavar value as an argument and return a result. They are passed to the parse_command_line() function as a dictionary keyed by the metavar they operate on.

For example, here is a set of metavar handlers for converting the START_DIRECTORY metavar into an absolute, normalised path. They can also set errors by raising a getopt.GetoptError exception.

import getopt
from commandtool import parse_html_config

def uniform_path(path):
    return os.path.normpath(os.path.abspath(path))

metavar_handlers = {
    'START_DIRECTORY': uniform_path,
    'CONFIG': parse_html_config
}

Basic Parsing and Program Handler

Now, let's do some basic parsing to see how the commands are used. We'll have one function called the program handler which will be responsible for dealing with any program options and, if necessary, calling the appropriate command handler to handle the options for the command chosen.

import sys

from commandtool import parse_command_line

def handle_program(
    metavar_handlers,
    command_handlers,
    option_sets,
    aliases,
    program_options,
    command_options,
    command,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

    Commands (aliases):
     name (n, nm)      find filenames containing a particular word
     content (c, ct)   find files whose content contains a particular word

    Try `%(program)s COMMAND --help' for help on a specific command."""
    # First, are they asking for program help?
    if program_options.has_key('help'):
        # if so provide it no matter what other options are given
        print strip_docsting(
            handle_program.__doc__ % {
                'program': os.path.split(sys.argv[0])[1],
            }
        )
        sys.exit(0)
    else:
        if not command:
            raise getopt.GetoptError("No command specified.")
        # Are they asking for command help:
        if command_options.has_key('help'):
            # if so provide it no matter what other options are given
            print strip_docsting(
                command_handlers[command].__doc__ % {
                    'program': os.path.split(sys.argv[0])[1],
                }
            )
            sys.exit(0)
        if program_options.has_key('verbose'):
            verbose_options = []
            for option in program_options['verbose']:
                verbose_options.append(option['name'])
            verbose_option_names = option_names_from_option_list(
                [option_sets['verbose'][0]]
            )
            quiet_option_names = option_names_from_option_list(
                [option_sets['verbose'][1]]
            )
            if len(verbose_options) > 1:
                raise getopt.GetoptError(
                    "Only specify one of %s"%(
                        ', '.join(verbose_options)
                    )
                )
            elif verbose_options[0] in quiet_option_names:
                logging.basicConfig(level=logging.ERROR)
            elif verbose_options[0] in verbose_option_names:
                logging.basicConfig(level=logging.DEBUG)
        else:
            logging.basicConfig(level=logging.INFO)
        # Now handle the command options and arguments
        command_handlers[command](metavar_handlers, option_sets, command_options, args)

if __name__ == '__main__':
    try:
        command = None
        prog_opts, command_opts, command, args = parse_command_line(
            option_sets,
            aliases,
            metavar_handlers = metavar_handlers,
        )
        handle_program(
            metavar_handlers,
            command_handlers,
            option_sets,
            aliases,
            prog_opts,
            command_opts,
            command,
            args
        )
    except getopt.GetoptError, err:
        # print help information and exit:
        print str(err)
        if command:
            print "Try `%(program)s %(command)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
                'command': command,
            }
        else:
            print "Try `%(program)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
            }
        sys.exit(2)

Notice that because the call to handle_program() is within a try... except block, any of the option handlers, command handlers or any other code can raise a getopt.GetoptError exception which will result in the error being printed to the screen and a short message printed about how to get help to use the correct options.

Dealing with Config Files

In addition to all the command options, I want users to be able to specify defaults in a config file. If a command doesn't recieve a neccessary command on the command line it can load it from the config file.

The metavars used to label the values associated with options on the command line earlier can also be used as keys in the config file as if the option that specifies the metavar has been added on the command line.

This merging doesn't happen automatically though, if a command handler wants to use the config file it handles the merging itself.

You'll see one of the option sets is for the internal variable config which sets the metavar CONFIG. When combined with a handler which parses the filename specified by the metavar CONFIG, the options can be automatically loaded if the -c or --config options are used.

In the command handler you can then do something like this to set a default value for the a metavar if it isn't specified on the command line.

def handle_name(options, command_options, args):
    ...
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    ...
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    ...

Notice that whether the value of START_DIRECTORY is extracted from the command line or on the config file, it is still run through the handler so the rest of the application beyond the handle_name() function remains the same.

Command Handlers

Now let's look at the command handlers. Their job is to look at the options used for each internal variable and call the appropriate Python API function to generate a result. They must then format the result for output on the command line.

Here's what they look like:

from commandtool import strip_docsting
from commandtool import option_names_from_option_list
from commandtool import set_error_on

def handle_command_name(
    metavar_handlers,
    option_sets,
    command_options,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] name [OPTIONS] LETTERS

    search DIRECTORY for a filename matching LETTERS

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type `%(program)s --help'
    """
    set_error_on(
        command_options,
        allowed=['config', 'start_directory']
    )
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = \
           metavar_handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    for found in name(**internal_vars):
        print found

def handle_command_content(
    metavar_handlers,
    option_sets,
    command_options,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] content [OPTIONS] LETTERS

    search DIRECTORY for a files containing LETTERS in their content

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -t, --file-type=FILE_TYPE                the file types to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type `%(program)s --help'"""
    set_error_on(
        command_options,
        allowed=['config', 'start_directory', 'file_type']
    )
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = metavar_handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    if command_options.has_key('file_type'):
        internal_vars['file_type'] = \
           command_options['file_type'][0]['value']
    for found in content(**internal_vars):
        print found

command_handlers = {
    'name': handle_command_name,
    'content': handle_command_content,
}

Notice that the docstring for each forms the help text for that command. In a real world applicaiton this approach is much simpler than trying to automatically generate help options. In this example, display of help text for the command is actually handled in handle_program() but if you wanted to be more consistent you could handle it in one of the command handlers.

Within each command handler you can choose to set an error if unexpected options are set. There is a helper function called set_error_on() to help you with this. It will set an error if any options have been set for internal variables you weren't expecting. It is used like this:

set_error_on(
    command_options,
    allowed=['config', 'site_directory', 'file_type'],
)

Summary of the Processing Pipeline

With these features the following actions happen:

  • Prepare option_sets, aliases and option_handlers
  • Pass them to the parse_command_line() function to have them parsed
  • If the options specified aren't valid an error messages is displayed
  • If successful and there are some command options, the options are passed through any required handlers to format the values specified
  • Otherwise the options are organised into the four variables: program_options, command_options, command, args. The program_options and command_options are dictionaries with the internal variables as their key and a list of all the options which affect that variable as the value. The items in each option list are actually dictionaries with other useful information about the option, such as its position on the original command line, the metavar, help text etc.
  • This parsed data can then be sent to a command handler which checks the options are valid for the particular command, sets the internal values it requires based on the options, calls the real Python API with the internal variables and formats the result for display on the command line.

Testing the Example

Let's test the basic help options:

$ python example.py
No command specified.
Try `example.py --help' for more information.
$ python example.py --help
usage: example.py [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

Commands (aliases):
 name (n, nm)      find filenames containing a particular word
 content (c, ct)   find files whose content contains a particular word

Try `example.py COMMAND --help' for help on a specific command.
$ python example.py name
Expected exactly one argument, LETTERS
Try `example.py name --help' for more information.
$ python example.py name --help
usage: example.py [PROGRAM_OPTIONS] name [OPTIONS] LETTERS

search DIRECTORY for a filename matching LETTERS

Options:
 -s, --start-directory=START_DIRECTORY    the directory to search
 -c, --config=CONFIG                      the config file to use

For PROGRAM_OPTIONS type `example.py --help'

Let's check that putting --help in front of the command name triggers the program help, not the name command help:

$ python example.py --help name
usage: example.py [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

Commands (aliases):
 name (n, nm)      find filenames containing a particular word
 content (c, ct)   find files whose content contains a particular word

Try `example.py COMMAND --help' for help on a specific command.

It does.

So far so good, now let's try the command out for real.

To test the example let's set up a sample directory to search:

$ mkdir start_here
$ cd start_here
$ cat << EOF >> one.txt
This is a sample file which contains the text 'one'.
EOF
$ cat << EOF >> two.txt
This is a sample file which contains the text 'two'.
EOF
$ cat << EOF >> two.py
# This is a Python file which contains the text 'two'.
EOF
$ cd ../

Now let's find all files with two in their names within the start_here directory:

$ python example.py name -s start_here
Expected exactly one argument, LETTERS
Try `example.py name --help' for more information.

Oops, forgot the letters:

$ python example.py name -s start_here two
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

Great! It has found the files. How about using --start-directory instead of -s:

$ python example.py name --start-directory start_here two
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

No trouble. What about trying to confuse it by changing the order?

$ python example.py name two --start-directory start_here
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

Still works.

Now let's try to search the contents of the files with the ct alias for the content command:

$ python example.py ct -s start_here two
INFO:find:Finding '.txt' files in '/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.txt

Yep, that's correct because it only searches .txt files by default, let's specify .py files instead and put the option in completely the wrong place:

$ python example.py -t .py ct two -s start_here
INFO:find:Finding '.py' files in u'/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.py

No trouble.

Now let's introduce an html config file:

$ cat << EOF >> test.html
<html>
<head><title>Example Config File</title></head>
<body>
<h1>Example Config File</h1>
<table class="commandtool-config">
<tr><td class="metavar"><tt>START_DIRECTORY</tt></td>
    <td>The directory to begin the search</td>
    <td>/home/james/Desktop/start_here</td></tr>
</table>
</body>
</html>
EOF

You'll need to update the path to point to your start_here directory. For this example, make sure test.html is in the same directory as example.py, not in the start_here directory.

Now let's use this and not specify the -s option to see if it correctly gets the value from the config file:

$ python example.py -t .py ct two --config test.html
INFO:find:Finding '.py' files in u'/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.py

No problem, config file parsing works too.

Now let's test the program options --verbose:

$ python example.py --verbose -t .py ct two --config test.html INFO:find:Finding '.py' files in u'/home/james/Desktop/start_here' which contain the letters 'two' DEBUG:find:Trying u'two.py' DEBUG:find:Matched u'/home/james/Desktop/start_here/two.py' /home/james/Desktop/start_here/two.py

As you can see, DEBUG messages get printed in addition to INFO messages. Now with --quiet:

$ python example.py --quiet -t .py ct two --config test.html /home/james/Desktop/start_here/two.py

No messages get printed at all. Let's check we can put these program options after the command:

$ python example.py -t .py ct two --config test.html -q
/home/james/Desktop/start_here/two.py

Yes we can. What happens if we specify them both together:

$ python example.py -t .py ct two --config test.html -qv
Only specify one of -q, -v
Try `example.py content --help' for more information.

As you can see the comman line API is very flexible.

Dealing with Man Pages

The man pages can be automatically generated from the docstring of the command line functions as long as you write them properly. See the man pages from Python post for setting up the rst2man.py tool.

You can then copy and paste the docstrings from the command handlers to popualte the man page.

Creating a Script

Rather than requiring users to run python example.py you can create a script which handles the command line arguments instead so that users can just run example. Here's an example script:

#!/usr/bin/env python

import sys
import getopt
sys.path.append('/home/james/Desktop')
from example import *

if __name__ == '__main__':
    try:
        command = None
        prog_opts, command_opts, command, args = parse_command_line(
            option_sets,
            aliases,
            metavar_handlers = metavar_handlers,
        )
        handle_program(
            metavar_handlers,
            command_handlers,
            option_sets,
            aliases,
            prog_opts,
            command_opts,
            command,
            args
        )
    except getopt.GetoptError, err:
        # print help information and exit:
        print str(err)
        if command:
            print "Try `%(program)s %(command)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
                'command': command,
            }
        else:
            print "Try `%(program)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
            }
        sys.exit(2)

You'll need to ensure that the version of Python at the top of the file has access to both the example.py script and the commadtool module for this to work. You can do so by installing them but even easier is to modify /home/james/Desktop to point to the place you've stored those files. You can then set its permissions to 755:

$ chmod 755 example

and execute it like this:

$ ./example --help
usage: example [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

Commands (aliases):
 name (n, nm)      find filenames containing a particular word
 content (c, ct)   find files whose content contains a particular word

Try `example COMMAND --help' for help on a specific command.

If you prefer you can move it to /usr/bin then you can run example from anywhere. Make sure there isn't already a file there called example though.

$ sudo cp example /usr/bin
$ example --help
usage: example [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

Commands (aliases):
 name (n, nm)      find filenames containing a particular word
 content (c, ct)   find files whose content contains a particular word

Try `example COMMAND --help' for help on a specific command.

Update 2009-05-25: Thanks to John Cavanaugh for pointing out these alternatives. John is actively investigating the second:

Cmd2
Oriented for console applications rather than just a cmdline tool
Cmdln
Probably the most mature
Subcommand
Attempts to parse a usage string to create the subcommands/options

(view source)

James Gardner: Home > Blog > 2009 > Writing a Python Command Line...