Chapter 17: Pylons’ Internal Architecture

Entry Points and websetup.py

If you look at the setup.py file for a Pylons project, this is what the entry point definition looks like:

setup(
    ...
    entry_points="""
    [paste.app_factory]
    main = simplesite.config.middleware:make_app

    [paste.app_install]
    main = pylons.util:PylonsInstaller
    """,
)

This means your Pylons application implements the functionality expected of an application supporting the main entry point of a paste.app_factory group and the main entry point for the paste.app_install group. In effect, your Pylons application behaves like a Paste app factory and as a Paste installer.

You’ll see how the main entry point of a paste.app_factory group is used in the next section, so for the moment, let’s concentrate on the main entry point for the paste.app_install group, which is used by the paster setup-app command.

When you run the paster setup-app command, Paste Script discovers the Pylons application you want to set up by looking at the use line in the [app:main] section of the config file. It looks like this:

[app:main]
use = egg:SimpleSite

The Pylons config file is in a special format understood by the PasteDeploy package. The use line simply tells Paste Deploy which package to use and that the package is an egg. The use line can also be written like this:

[app:main]
use = egg:SimpleSite#main

This tells Paste Deploy that the entry point it is looking for is called main, but this is assumed to be the case anyway if the #main part is ignored.

Once paster setup-app knows the package to look for and the name of the entry point, it looks it up in the paste.app_install group with code similar to this:

from pkg_resources import load_entry_point
entry_point = load_entry_point('SimpleSite', 'paste.app_install', 'main')
installer = entry_point.load()

The load_entry_point() function can also take a version specification such as 'SimpleSite>=0.1.0' as its first argument.

Now, Pylons applications are unusual in the sense that this entry point doesn’t point to an object in the Pylons project in which it is defined but in pylons.util instead. This means the object that is loaded is always pylons.util.PylonsInstaller no matter which Pylons project is being used. The PylonsInstaller object itself is responsible for then calling the setup_app() function in your project’s websetup.py file.

Now that you have seen how entry points can be used to set up a Pylons application, you can learn about how they are used when serving an application.

Default Config Options

The first section in a Pylons config file is the [DEFAULT] section. This contains a set of variables that will make up the global configuration, accessed in a Pylons application as config.global_conf. The options are also passed to each of the functions used to construct objects in other sections of the config file. These options are labeled DEFAULT rather than global_conf because they also work as if they were present in each of the other sections (apart from the logging sections). As an example, consider how the debug option works. In the [DEFAULT] section, the debug option is set to true. This means that in the [app:main] section, if no debug option is specified, debug in that section will also be set to true. In effect, options in the [DEFAULT] section are providing defaults for the other sections. When the debug option is set to false in the [app:main] section, the value in the [DEFAULT] section is overridden, and debugging is disabled.

Now that you understand the role of the [DEFAULT] section, let’s see how Paste Deploy handles the other sections.

Constructing a Server

When Paste Deploy parses a config file, the [server:main] section is inspected to find out which server to use. The Pylons config file typically has a section that looks like this:

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000

The use line tells the Paste Script package’s server plug-in that it should look up an egg entry point in the Paste package named http. Because this is a [server:main] section, Paste Script knows that the entry point will be in the paste.server_runner entry point group.

The host option tells the server specified in the use line which host to serve the application on. You’ll remember from Chapter 3 that if you want to serve an application on all interfaces, you will need to change the host option to 0.0.0.0 because it is set by default to 127.0.0.1 to prevent you from accidentally serving an application across a network when it is running in debug mode on your development machine. You can also specify a hostname or domain name as long as they can be correctly resolved on your computer.

The port option simply tells the server which port to serve on. For production use, you would use port 80, but for development use, port 5000 is fine.

Let’s take a closer look at how Paste Deploy loads the Paste HTTP server from the use line. First the Paste package has the following as part of its entry point definition:

entry_points="""
    ...
    [paste.server_runner]
    http = paste.httpserver:server_runner
    ...
"""

The use = egg:Paste#http line therefore points to the server_runner() function in the paste.httpserver module. This function is responsible for taking the arguments Paste Deploy sends it from the information it parsed from the config file and returning a running server. The function looks like this:

def server_runner(wsgi_app, global_conf, **kwargs):
    ...

The server_runner() function takes a dictionary of all the options in the [DEFAULT] section of the config file as the global_conf argument and all the options specified in [server:main] as keyword arguments that are gathered up by **kwargs into a dictionary. In this case, the global_conf options aren’t used, but the port and host options passed as keyword arguments are. The wsgi_app argument is the WSGI application to serve. It is based on the other objects specified in the config file. I’ll cover these in the next sections.

Once all the options are in place, the function is called, and this starts the server.

Constructing an Application

Constructing the application happens using a similar mechanism, except Paste Deploy starts by looking for either an [app:main] section, a [composite:main] section, or a [pipeline:main] section in the config file. Only one section other than [server:main] can have the name :main. I’ll cover composite applications and pipelines later in the chapter, so let’s concentrate on what happens when your main section is an [app:main] section.

The Pylons config file [app:main] section starts like this:

[app:main]
use = egg:SimpleSite
...

Paste Deploy knows to look for paste.app_factory entry point group to load an application factory. As was the case earlier in the chapter, because no #name is specified after the definition, Paste Deploy will look for an entry point named main.

Let’s look at SimpleSite’s setup.py file again to see whether it contains a paste.app_factory group:

setup(
    ...
    entry_points="""
    [paste.app_factory]
    main = simplesite.config.middleware:make_app

    [paste.app_install]
    main = pylons.util:PylonsInstaller
    """,
)

In this case, the main entry point in the paste.app_factory group points to the make_app() function in simplesite.config.middleware, so it is this function that is responsible for assembling the Pylons application and middleware that will form the WSGI application and middleware.

The make_app() function looks like this:

def make_app(global_conf, full_stack=True, **app_conf):
    ...
    return app

Once again, the config options from the [DEFAULT] section get passed to the function as global_conf, just as they did when the server was being called, and once again the section-specific options get passed as named parameters that this time are gathered up into the app_conf dictionary.

The WSGI application returned by make_app() as app is what PasteScript’s serve plug-in passes as the first argument to the server_runner() function when it serves the application.

We’ll take a detailed look at what happens in the make_app() function to turn the configuration options into a Pylons application later in the chapter, but there is one point worth noting first. Because the full_stack option is specified as an argument to make_app(), it doesn’t get added to the app_conf dictionary in the same way as all the other variables. There isn’t really a good reason for this, so it might change in a future version of Pylons to be more like this:

def make_app(global_conf, **app_conf):
    full_stack = asbool(app_conf.get('full_stack', 'true'))
    ...
    return app

As you’ve probably realized, the Pylons config file format is slightly more powerful than most frameworks’ config files. It turns out that you can actually use the config file to directly assemble a whole range of WSGI (and hence Pylons) applications into one composite application, and you can even add middleware to individual WSGI (or Pylons) applications using filters and pipelines. Let’s start with composite applications.

Composite Applications

Composite applications are WSGI applications that are made up of other WSGI applications. A good example of a composite app is a URL mapper that mounts WSGI applications at different paths relative to the base URL. Here’s an example that mounts your Pylons application at the path /pylons and mounts an application for handling downloads at the path /downloads:

[composite:main]
use = egg:Paste#urlmap
/pylons = pylons
/downloads = staticapp

[app:staticapp]
use = egg:Paste#static
document_root = /path/to/docroot

[app:pylons]
# The standard [app:main] section goes here
use = egg:SimpleSite
... etc

When this config file is loaded, Paste Deploy will look for the [server:main] section and any other main section and find that this time, rather than the second main section being an [app:main] section, it is a [composite:main] section. It will therefore look up the urlmap entry point name in the Paste package from the paste.composite_app group, and this in turn will load the paste.urlmap.urlmap_factory() to load a paste.urlmap.URLMap. The URLMap will be set up with the two WSGI applications (the Pylons app and the static app) at the appropriate paths.

Notice that for this to work the Pylons configuration section had to be renamed from [app:main] to [app:pylons]. If you hadn’t done this, the two main sections would have conflicted.

One reason for setting up other WSGI applications via a composite section in the config file rather than as a Pylons controller or from a Pylons controller action is that none of the Pylons middleware will be in place. This is useful if you find that the Pylons middleware interferes with some aspect of the behavior of the other WSGI app when it is mounted inside a Pylons application.

It is worth noting that the Paste URLMap in this example will automatically adjust the SCRIPT_NAME and PATH_INFO environment variables so that your WSGI applications mounted under it will be able to work out where they are. This means that as long as the applications you mount use the h.url_for() helper (or their own equivalents), then the URLs generated by the WSGI applications under URLMap will still be correct even though they are mounted at a path other than /.

Pipelines and Filters

Now let’s look at pipelines and filters. Filters are just Paste Deploy’s name for functions that set up WSGI middleware, and pipelines are just Paste Deploy’s name for a middleware chain.

To use filters, instead of specifying the main section to be your Pylons application or a composite application, you specify a pipeline. The pipeline takes just one configuration option, which is a list of the filters to use. The list should always end with the WSGI application you want to serve, which is usually your Pylons application.

Let’s use a pipeline to add the Gzip middleware you developed in the previous chapter. Remember that you added it to the simplesite.lib.middleware module you created. Here’s an example of how the relevant sections of the config file should be set up:

[pipeline:main]
pipeline = gzip pylons

[filter:gzip]
use = egg:SimpleSite#gzip

[app:pylons]
# Your normal Pylons [app:main] section
use = egg:SimpleSite
...

Once again, because you’ve named the pipeline main, the section holding the configuration for Pylons will have to be renamed from [app:main] to something else, and once again this example uses [app:pylons] as a sensible name, so the WSGI application at the end of the pipeline list is named pylons to point to that section.

[pipeline:main]
pipeline = filter1
           filter2
           filter3
           app

There is just one problem with this example as it stands: the gzip entry point doesn’t exist in the SimpleSite application yet, so this example won’t be able to actually load your Gzip middleware. You could fix this by using the Paste Gzip middleware instead by using use = egg:Paste#gzip, but let’s update the SimpleSite project so that your middleware can be used. To do that, you need to know about factories.

Understanding Factories

In the examples so far, the use option has been used to specify a named entry point within an entry point group in a package, and that entry point has been used to load a function that, when called with arguments reflecting config file options, results in the construction of a particular object such as a server or WSGI application. In Paste Deploy terminology, the function pointed to by the entry point is known as a factory because it produces the desired object.

As you’ve seen, different types of sections in the config file use different entry point groups, and these point to different types of factories. Server sections point to server factories, app sections point to application factories, composite sections point to composite factories, and filters point to filter factories. Factories effectively translate the config options passed to them into appropriate variables that can be used to construct the objects they create. For the Gzip middleware to be usable directly from the config file, you’ll need to create a filter app factory for it. A suitable factory would look like this:

def make_gzip_middleware(app, global_conf, **app_conf):
    compresslevel = int(app_conf.get('compresslevel', 9))
    return GzipMiddleware(app, compresslevel)

This factory would be passed the options in [DEFAULT] as the global_conf argument and the options in the section for that filter as keyword arguments that can be gathered up into the app_conf dictionary. The app is a WSGI application that the middleware should wrap. As you can see, the compresslevel argument (a number from 0–9 is used to specify how much compression should be applied) is turned into an integer if it is present, and if not, the default value of 9 is used. The WSGI application is then wrapped in the middleware and returned.

Add this factory to the SimpleSite lib/middleware.py file you created in the previous chapter if you want to test the example.

Now that the factory is in place, let’s set up the entry point. Filter factories are put in the paste.filter_app_factory entry point group. Edit the SimpleSite setup.py file to update the entry points section:

setup(
    ...
    entry_points="""
    [paste.app_factory]
    main = simplesite.config.middleware:make_app

    [paste.app_install]
    main = pylons.util:PylonsInstaller

    [paste.filter_app_factory]
    gzip = simplesite.lib.middleware:make_gzip_middleware
    """,
)

You’ll need to run python setup.py develop again for the entry point change to be noticed; otherwise, you’ll see this error:

LookupError: Entry point 'gzip' not found in egg 'SimpleSite' (dir: /home/james/Desktop/SimpleSite2b; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: )

Once you’ve reinstalled the application, you will be able to test the filter. Be sure to remove the app = GzipMiddleware(app, 5) line from config/middleware.py; otherwise, you will get a Content-Encoding header with gzip specified twice. The browser won’t understand this and will expect normal CSS or JavaScript instead of compressed content and will therefore most likely complain about illegal characters.

Once the Gzip middleware is disabled in config/middleware.py, update the config file. The lines 7-13 are the ones that have been changed:

::

linenothreshold

5

...
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000

[pipeline:main]
pipeline = gzip pylons

[filter:gzip]
use = egg:SimpleSite#gzip

[app:pylons]
use = egg:SimpleSite
full_stack = true
cache_dir = %(here)s/data
beaker.session.key = simplesite
beaker.session.secret = somesecret
...

::

linenothreshold

5000

With these changes in place, if you test the SimpleSite application and use LiveHTTPHeaders or Firebug to inspect the response at a URL such as http://localhost:5000/page/edit/6, you’ll see the .js and .css files are still being Gzipped.

There is also an alternative syntax you can use if you simply want to wrap a WSGI application in one piece of middleware. You can use the filter-with option. Here’s the config file with the [pipeline:main] section removed, the [app:main] section reintroduced, and the filter-with option being used. Lines 7 and 9 have changed:

::

linenothreshold

5

...
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000

[app:main]
use = egg:SimpleSite
filter-with = egg:SimpleSite#gzip
full_stack = true
cache_dir = %(here)s/data
beaker.session.key = simplesite
beaker.session.secret = somesecret
...

::

linenothreshold

5000

If you test this setup, you’ll find it behaves in the same way as before and that the CSS and JavaScript files are Gzipped correctly.

There is one complication with the setup described so far. Paste Deploy, and hence the Pylons config file, supports two types of filters. The one you are using here is called a filter-app because the WSGI application is passed along with the configuration options to the filter-app factory. Another type of factory is one that is passed only the configuration options and that returns a function that, when called, returns the WSGI application wrapped in the filter. Such a factory is called a filter and would look like this:

def make_gzip_middleware_filter(global_conf, **app_conf):
    compresslevel = int(app_conf.get('compresslevel', 9))
    def filter(app):
        return GzipMiddleware(app, compresslevel)
    return filter

Add this to the lib/middleware.py file too, and update the entry points in setup.py (lines 13-14):

::

linenothreshold

5

setup(
    ...
    entry_points="""
    [paste.app_factory]
    main = simplesite.config.middleware:make_app

    [paste.app_install]
    main = pylons.util:PylonsInstaller

    [paste.filter_app_factory]
    gzip = simplesite.lib.middleware:make_gzip_middleware

    [paste.filter_factory]
    gzip = simplesite.lib.middleware:make_gzip_middleware_filter
    """,
)

::

linenothreshold

5000

If you run python setup.py develop again, the new entry points will take effect. When you have both a filter and a filter-app for the same entry point name, Paste Deploy uses the filter, so the new function you’ve added will be used in preference to the previous one.

Alternative Ways of Specifying Factories

So far in this chapter you’ve seen two ways to specify a factory. The first is to use an egg URI such as the following one to identify a factory via an entry point:

[filter:gzip]
use = egg:SimpleSite#gzip

The second approach is to refer to a different section within a config file and have that section be responsible for looking up the factory. You’ve just seen this in the discussion of pipelines where the items in the pipelines were the names of sections, for example:

[pipeline:main]
use = gzip pylons

In addition to these techniques, there are two other approaches. The first is to specify a section in an entirely different config file like this:

[app:main]
use = config:development.ini#main

You’ve actually seen this technique in Chapter 12 when it was used to ensure the test setup used the same configuration as the development setup. In that particular situation, you’ll recall that the approach also had unexpected consequences because the websetup.py setup_app() function was automatically called each time you ran the tests and this interfered with your development database setup.

The final way to specify a factory is to point to it in some Python code:

[filter:gzip]
paste.filter_app_factory = simplesite.lib.middleware:make_gzip_middleware

In this last approach, rather than using the word use, you have to specify the entry point group name as the option name. This might seem slightly counterintuitive, but it is how the config file format works.

Configuration Inheritance

Paste Deploy also supports a very simple form of inheritance so that if you have to specify multiple sections with similar configuration options, you don’t have to write them all out again. Here’s how it works:

[app:main]
use = egg:AnimalTracker
cat = Tabby
dog = St Bernard

[app:other]
use = main
dog = Jack Russell

In this example, the other app inherits all the options from the main app, including the value for the cat option, but it overrides the dog option with the value Jack Russell.

The Pylons config file is actually a very flexible format. You can learn more about it at http://pythonpaste.org/deploy/.

Accessing the Server, Application, and Filters

Paste Deploy provides three functions to allow you to access servers, applications (pipeline, composite, and app sections), and filters:

loadserver(uri, name=None, **kw)

This function returns a server wrapper function that takes the WSGI application to serve as its first argument. When called, the function uses the options from the [server:main] section, together with the WSGI application passed to it, to start the server. If your config file defines multiple server sections, you can specify the one to use with name. For example, here you load the alternative server instead of the main one:

from paste.deploy import loadserver
server_wrapper = loadserver('config:/path/to/config.ini', name='alternative')
# Serve the application with the options in ``[server:alternative]``
server_wrapper(app)

loadapp(uri, name=None, **kw)

This is the function you are most likely to use, which returns a WSGI app based on the name of the application section you specify. If you don’t specify a section name, it assumes you want the main section so loads the application based on the information in the [app:main], [composite:main], or [pipeline:main] section in the config file, depending on which you’ve used. For example:

from paste.deploy import loadapp
app = loadapp('config:/path/to/config.ini')

loadfilter(uri, name=None, **kw)

This function behaves similarly to loadserver() returning a wrapper function that, when called with a WSGI application, returns the application wrapped in the middleware specified in the filter section named name and constructed with the options from that section.

This doesn’t load filter-app factories, just filter factories.

from paste.deploy import loadfilter
filter_wrapper = loadfilter('config:/path/to/config.ini')
# Wrap the application with the middleware specified by ``[filter:main]``
app = filter_wrapper(app)

Notice that each of these functions takes a config URI, not simply a path to a config file as you might have expected. Each of these functions also takes a relative_to argument that you can use if you want to specify a relative location for the URI; you can use it like this:

import os
from paste.deploy import loadserver
server = loadserver('config:config.ini', relative_to=os.getcwd())
# Start the server
server(wsgi_app)

If you try to use a relative path without specifying relative_to, you will get an error explaining no context keyword argument given. This error isn’t particularly informative, but specifying relative_to resolves this error.

Accessing Configuration Options

There are occasions when even the functions described so far aren’t low level enough and you want to get directly at the individual options set. For this circumstance, Paste Deploy provides an appconfig() function that returns a config object. It has the following definition:

def appconfig(uri, name=None, relative_to=None, global_conf=None)

The config object returned has two attributes, local_conf and global_conf, both of which are dictionaries. The .global_conf dictionary contains all the options from the [DEFAULT] section, and the .local_conf dictionary contains all the options from whichever application section you specified as the name argument to appconfig() defaulting to main if no name is specified.

The config object itself behaves like a dictionary too. It has all the keys of both the local_conf dictionary and the global_conf dictionary, but where two keys have different values, the local_conf value overrides the global conf value. The config object therefore has the same values you would access as pylons.config['app_conf'] and the .global_conf attribute contains the same values you would access as pylons.config['global_conf'].

Here’s an example demonstrating this:

>>> from paste.deploy import appconfig
>>> config = appconfig('config:/path/to/config.ini')
>>> combined = {}
>>> combined.update(config.global_conf)
>>> combined.update(config.local_conf)
>>> print config.items() == combined.items()
True

As you can see, the configuration in the config object itself matches that obtained by combining the global_conf and local_conf dictionaries.

Creating a Pylons Application with Paste Deploy

Using the Paste Deploy loadapp() function described earlier is the best way of getting access to the Pylons WSGI application. At the start of this chapter, you learned how to use entry points to access the make_app() function in your project’s config/middleware.py file. This function is a standard Paste Deploy app factory, so although you can create a Pylons WSGI application object by calling it directly with the configuration options you want to create the application with, it is usually much easier to use the loadapp() function. This function does both steps at once, looking up the entry point and calling the make_app() factory with the configuration options from the config file.

You can get the WSGI application object from your Pylons configuration file like this:

from paste.deploy import loadapp
wsgi_app = loadapp('config:/path/to/simplesite/development.ini')

You can then serve the file using a WSGI server. Here is an example using the WSGI Reference Implementation server included in Python 2.5 and newer:

from paste.deploy import loadapp
wsgi_app = loadapp('config:/path/to/config.ini')

from wsgiref import simple_server
httpd = simple_server.WSGIServer(('',8000), simple_server.WSGIRequestHandler)
httpd.set_app(wsgi_app)
httpd.serve_forever()

The paster serve command you are used to using while developing Pylons projects combines these two steps of creating a WSGI app from the config file and serving the resulting file to give the illusion that it is serving the config file directly. You can test the example by visiting http://localhost:8000.

Application State vs. Request State

Once the application has been created with all the middleware components correctly set up, it is held in memory by the server, ready to receive a request. The application object, the middleware, and all the other Pylons objects that were configured when make_app() was called will persist between each request. This means the make_app() factory is called only once when the Pylons application is created, but the app object returned from make_app() is called on each request. This makes Pylons very fast because the app object doesn’t need to be re-created to handle a request, but it also means that the process of creating a Pylons application is completely separate from the process of handling a request.

In the next sections, you’ll see in detail which processes occur when the app object is being set up and which occur before and after a request has been handled by a controller action.

Loading the Pylons Environment

The first thing that happens in make_app() is a call to load the environment:

# Configure the Pylons environment
load_environment(global_conf, app_conf)

This function is actually located in your project’s config/environment.py file and is responsible for any configuration of your application that you don’t want to expose to the end user of an application in the config file. You have to be slightly careful when editing config/environment.py because some aspects of Pylons’ behavior rely on the objects that are set up there.

The load_environment() function is responsible for the following:

• Setting up the paths that will be used in your project to access controllers, static files, and templates. By default, the paths are configured to point to your project’s controllers, public, and templates directories, respectively.

• Initializing the Pylons config object with information about the project including where the function to set up the routes is, which module should be used as the h object within templates, and which object should be used as the app_globals object. By default, these are set to the config/routing.py file’s make_map() function, the project’s lib/helpers.py module, and an instance of the Globals class in lib/app_globals.py, respectively.

• Using the config object to indirectly attach a mako_lookup attribute to the app_globals object, which will later be used by the render() function to render Mako templates. The mako_lookup attribute is actually an instance of a Mako TemplateLookup object, and as you’ll recall from Chapter 5, this can be customized to change Mako’s behavior or replaced if support for a different templating language is required.

• Setting up a SQLAlchemy engine and initializing the model (if you chose a project template that included SQLAlchemy when running the paster create command). The way the engine is created and the role of init_engine() is explained in Chapter 7.

At the end of the load_environment() function are these lines:

# CONFIGURATION OPTIONS HERE (note: all config options will override
# any Pylons config options)

Any further customization you want to do should usually happen after these lines.

The PylonsApp Instance

After the environment has been loaded to set up the app_globals, h, and config objects as well as the template engine and SQLAlchemy (if necessary), a Pylons application is created. This is done with the following lines in make_app():

# The Pylons WSGI app
app = PylonsApp()

The PylonsApp instance is the very heart of your Pylons application; it is defined in the pylons.wsgiapp module, and its instance (app) is a valid WSGI application. At this stage, all app has to do is initialize itself with the config object, the app_globals object, and the package name and various options that will be used to set up the request and response during a request. This all happens behind the scenes and is not something you would normally have to deal with. You’ll hear more about the PylonsApp instance when I discuss how it behaves during a request.

The Middleware Chain

Next, the PylonsApp instance app is wrapped in a series of different middleware components, each of which is initialized:

# Routing/Session/Cache Middleware
app = RoutesMiddleware(app, config['routes.map'])
app = SessionMiddleware(app, config)
app = CacheMiddleware(app, config)

First the RoutesMiddleware is initialized and is passed the route map it will be expected to deal with via the config['routes.map'] object that was set up in the call to load_environment().

After the RoutesMiddleware is set up, SessionMiddleware and CacheMiddleware are both set up. These use the beaker.* configuration options from the config object in order to initialize themselves. The options specify where the session and cache stores should be set up. By default, this happens in a data directory relative to the config file being used to serve the application.

What happens next depends on whether you’ve set full_stack to true or false in the configuration file. If it is true, which is the default, then the ErrorHandler and StatusCodeRedirect middleware will be set up; otherwise, they’ll be ignored.

if asbool(full_stack):
    # Handle Python exceptions
    app = ErrorHandler(app, global_conf, **config['pylons.errorware'])

    # Display error documents for 401, 403, 404 status codes (and
    # 500 when debug is disabled)
    if asbool(config['debug']):
        app = StatusCodeRedirect(app)
    else:
        app = StatusCodeRedirect(app, [400, 401, 403, 404, 500])

If the fullstack option is enabled, the ErrorHandler is set up with options from config['pylons.errorware']. These in turn are set up when the config object’s init_app() method is called in config/environment.py and contain the options that will be used in error reporting such as the e-mail address to send error reports to, the SMTP server to use, and who the e-mail should appear to have come from. The values used are all obtained from the [DEFAULT] section of the config file. The subject of the e-mail and the error log to use are also passed to the ErrorHandler in the same dictionary, but these values cannot be configured in the config file.

If debug mode is on, the ErrorHandler middleware sets up the WebError package’s EvalException middleware, which provides the Pylons Interactive Debugger you first saw in Chapter 4; otherwise, it sets up the WebError package’s ErrorMiddleware to handle the e-mail error reports.

The StatusCodeRedirect middleware doesn't do anything particularly exciting during application initialization, it just wraps itself around the app object.

Next to be wrapped around the app object is the RegistryManager middleware. This doesn’t need any configuration as the application is being created, but its importance will become apparent when you look at how Pylons handles a request in the next section.

# Establish the Registry for this application
app = RegistryManager(app)

The final piece of middleware is the Cascade. This is configured with an instance of a StaticURLParser as its first argument. The StaticURLParser will be responsible for serving static files and is configured with config['pylons.paths']['static_files'], which was set up in config/environment.py to point to your project’s public directory. The second item in the list specified as an argument to the Cascade is the app object all the middleware components have been wrapping.

# Static files (If running in production, and Apache or another web
# server is handling this static content, remove the following 2 lines)
static_app = StaticURLParser(config['pylons.paths']['static_files'])
app = Cascade([static_app, app])
return app

At this point, the application state has been configured. All the middleware has been initialized, and the PylonsApp and StaticURLParser WSGI applications have been initialized. At this point, the app object is ready to handle a request, and this is where the middleware components play an important part.

With the application state successfully configured, make_app() can return the app object to the server, ready to be called to handle a request.

The Cascade

The first piece of middleware to be called on any request is the Cascade. When it is passed environ and start_response(), it first calls static_app to see whether the URL requested matches a file in your project’s public directory. If static_app returns a 404 Not Found response, the Cascade will instead call the main Pylons application.

The next middleware in the chain is the RegistryManager, which is responsible for managing the Pylons globals in a thread-safe way. Let’s look at it in detail before returning to the middleware chain.

The Registry Manager, StackedObjectProxy, and Pylons Globals

Pylons aims to avoid the use of any complex Python code that appears to behave in a “magical” way, but there is one aspect of the Pylons architecture that often appears to be magic to those who haven’t seen anything similar before: the Pylons globals. People wonder how a module global such as the Pylons template context c can be available throughout an application and yet take a different value depending on the thread in which it is executing.

The answer is that Pylons globals (such as c, request, and response) are all instances of the paste.registry.StackedObjectProxy class and not the objects they appear to be at all. When you access their properties or methods, they return the result of applying that action to the object they are associated with in the current thread so that they behave as if they were that object. Put another way, they proxy attributes to the correct object in a thread-safe way based on the scope of the request in which they run to give the illusion of being the objects themselves.

You can always access the underlying object for a particular global in the current request by calling the _current_obj() method on a StackedObjectProxy. For example, to obtain the actual request object being used in a request, rather than the StackedObjectProxy that is proxying to it, you could use pylons.request._current_obj().

Now that you know what a StackedObjectProxy is, let’s look at the role of the RegistryManager during a request. Here’s an example based on the Paste Registry documentation, which demonstrates how the globals are set up and used in a pure-WSGI environment:

from paste.registry import RegistryManager, StackedObjectProxy
from pylons.controllers.util import Request

request = StackedObjectProxy()

# WSGI app stack (imagine the middleware stack in config/middleware.py)
app = App()
app = RegistryManager(App)

# WSGI app (imagine the PylonsApp)
class App(object):
    def __call__(self, environ, start_response):
        obj = Request(environ)  # The request-like theread-local object you want
                                # to access via pylons.myglobal
        if environ.has_key('paste.registry'):
            environ['paste.registry'].register(request, obj)
            ...

The RegistryManager adds a new Registry() as environ['paste.registry'] on each request and calls the existing registry’s prepare() method to set it up for the new registry context.

The registration process involves telling the registry which objects in the current thread should be attached to which globals. This ensures that during the scope of the request, attributes from the correct real object will be returned any time you access an attribute or method of one of the Pylons global objects, even though the global is really a StackedObjectProxy.

You’ll see in the section “The Role of PylonsApp” that the registration happens within the PylonsApp instance app.

Returning to the Middleware Chain

Now that you have an idea of how the RegistryManager middleware works, let’s continue looking at how the other middleware components in the chain behave during a request.

If the full_stack option is set to true, the StatusCodeRedirect and ErrorHandler middleware are called. The StatusCodeRedirect middleware creates a copy of the request information, but it doesn’t do anything with it until it deals with the response after a controller action is called. Next, the CacheMiddleware and then the SessionMiddleware are called, each adding information to the environ dictionary that the PylonsApp app instance will use when it is eventually called to set up the Pylons cache and session functionality.

Next, the RoutesMiddleware is called. This now assembles the request URL from the information in the environ dictionary it is passed and works through the route map as you saw in Chapter 9. It adds the routing variables it matches to the environ dictionary as environ['wsgiorg.routing_args'].

Finally, the PylonsApp instance, app, gets called itself.

The Role of PylonsApp

The ultimate responsibility of app is to call a Pylons controller action and return a response, but it has to do a number of things first. Each of these is handled by a different method on the PylonsApp class, and they are called in the order described here:

PylonsApp.setup_app_env()

This creates all the global objects for the request including the Pylons request and response objects, the translator used for internationalization, and the Pylons template context global c. It adds these and other objects to the template context object as attributes and makes it available as pylons.pylons in the environ dictionary.

PylonsApp.register_globals()

If the RegistryManager middleware is present, each of the Pylons globals is registered with the registry. This includes pylons.response, pylons.request, pylons.app_globals, pylons.config, pylons.h, pylons.c, and pylons.translator. It will also register pylons.session and pylons.cache if the SessionMiddleware and CacheMiddleware are present. Pylons also adds a pylons.url object that is simply the current request URL.

At this point, all of these objects, when imported from the pylons module in your application, will proxy the attributes and methods you call on them to the correct underlying objects for the current request thread (that is, the ones created in the PylonsApp.setup_app_env() call or as the request passed through the middleware stack).

PylonsApp.load_test_env()

If Pylons is operating in testing mode, this method will be called to add the standard Pylons environment variables described earlier to environ['paste.testing_variables'] so that they are available as attributes of the paste.fixture response object you learned about in Chapter 12. These currently include the attributes req, response, tmpl_context and its alias c, app_globals and its alias g, h, and config. If the session and cache middleware are present, it also sets up session and cache attributes.

With all the objects correctly set up and registered, the dispatch to the controller action can begin. The following methods on PylonsApp are called in this order:

PylonsApp.resolve()

This uses the routing arguments that the RoutesMiddleware added to environ['wsgiorg.routing_args'] to retrieve a controller name and return the controller instance from the appropriate controller module.

PylonsApp.dispatch()

Finally, once the controller instance has been created, PylonsApp can call its __call__() method with environ and start_response() to begin the dispatch.

Even though the controller has now been dispatched, there is still some more you need to know about how the controller behaves to call the action.

The Role of WSGIController

You’ll remember that your project’s controllers are inherited from BaseController in your project’s lib/base.py file and that BaseController is inherited from pylons.controller.core.WSGIController. This means that when PylonsApp instantiates and calls one of your project’s controllers, it is actually calling the WSGIController.__call__() method.

The WSGIController.__call__() method proceeds as follows. First, it checks to see whether the controller has a __before__() method. If it does, it calls its _inspect_call() method that then calls _get_method_args() to find out which arguments the method expects. The __before__() method is then called with those arguments.

Next, the controller action itself needs to be called. This is done with a call to the controller’s _dispatch_call() method, which finds out which action to call from the routing variables and then calls __inspect_call() itself to find out the arguments the action needs and to call the action with those arguments.

Finally, after the action is called, the __call__() method checks to see whether an __after__() method exists. If it does, it will always be run on each request after the action, even if the action raises an exception or issues a redirect. Once again, __after__() is run by a call to the _inspect_call() method.

If an action is not found to handle the request, the controller will raise an “Action Not Found” error in debug mode; otherwise, a 404 Not Found error will be returned.

abort(), redirect_to(), and HTTPException

During each call to WSGIController’s _inspect_call() method when the request was being handled, the call to the method being inspected is wrapped in a try...``except`` block. If a certain type of exception known as an HTTPException occurs, Pylons will turn that exception into a normal HTTP response. This might seem like an odd behavior, but you’ll recall that Pylons provides the functions abort() and redirect_to() in pylons.controllers.util. These functions work by raising an HTTPException when they are called, so it is the code in the _inspect_call() method that is responsible for making these functions result in the correct HTTP response.

Since the _inspect_call() method is also used to call the __before__() and __after__() actions if they exist, abort() and redirect_to() can also be used in those actions.

Exception Handling

When an exception that isn’t an HTTPException is raised, it goes straight through the try...``except`` block in _inspect_call() and isn’t caught until another try...``except`` block in the error-handling middleware. If debug mode was enabled, the EvalException middleware would have been set up. If not, the ErrorMiddleware would be in place.

If the EvalException middleware is set up, you will see the familiar Pylons Interactive Debugger. If not, an error report gets e-mailed, and the exception is turned into a 500 Internal Server Error response by the ErrorMiddleware.

The StatusBasedRedirect middleware is below the error-handling middleware, so it receives the response after the error handler. At this point, the StatusCodeRedirect middleware can’t distinguish between a 500 error triggered by the ErrorMiddleware or a 500 response from a Pylons controller action, so both are treated in the same way. In debug mode 500, responses are ignored by the StatusCodeRedirect middleware; with debug mode disabled, an error document is displayed.

Error Documents

When the StatusCodeRedirect middleware receives a response that it was set up to intercept, it uses a copy of the request information (which it made when the request was passing through it toward the controller action) to start a subrequest to the error controller’s document() action to generate the familiar Pylons error page. You’ll see more about error documents, how they work, and how to customize them in Chapter 19.

Streaming Content

If you need to stream content, you can always use one of the techniques described in Chapter 16 to mount a WSGI application in a Pylons application and set up the WSGI application to stream the content, but Pylons controller actions support streaming directly too.

You need to turn debug mode off to stream content because the exception middleware needs to have the entire response finished before it will pass on the result farther down the middleware chain.

Turn off debug mode in your INI file by uncommenting the following line:

#set debug = false

Now that it’s disabled, returning an iterator object or a generator is sufficient to start streaming output. Here’s an example that sends a new number every two seconds:

import time

class StreamingController(object):

    def output(self):
        def output_pause():
            number = 0
            while 1:
                number += 1
                yield number
                time.sleep(2)
        return output_pause()

If you were to set this up in a Pylons project and visit http://localhost:5000/streaming/output, you should now see a number sent every two seconds, continuing forever. If not, ensure you have set debug to false in the config file.

Returning Unicode from an Action

Now that you’ve seen some of the less usual cases for controller action responses, let’s look at what happens during a normal response when you return a string, most likely as a result of a call to render() to render a template.

First the response passes back from the WSGIController.__call__() method and the PylonsApp.dispatch() method to the middleware chain. The response will start at the middleware defined at the top of config/middleware.py and end with the ones defined at the bottom. As it passes through the SessionMiddleware, it will have a session cookie added to it if there was a session cookie in the request or if you called session.save() during the request to create a session store for the current user. The response will then pass back through the other middleware without being altered and eventually be sent back to the server where it is passed to the browser.