Monthly Archive for September, 2010

wsgi middlewares for profiling and debugging

A while back I implemented a debugging and profiling middleware, and I’ve been using those with Pylons recently. I think their pretty useful, so I’ve wrapped them up into an installable egg that contains Paste interfaces in setup.py. This allows you to easily insert the middleware into any existing Paste project (e.g. a Pylons project). I’m basically going to use this middlewares project as a dumping ground for middleware I find useful. It currently contains a debug (via DBGP) middleware, a profiler and a csrf middleware. There is nothing that says you must use Paste with these, Paste just makes it easier. The csrf middleware is currently tied to using Beaker.

Here’s a screenshot of Komodo IDE debugging my Pylons app. Lots of other debuggers support DBGP, but for many reasons I like Komodo.

Below is a partial output of line profiling the csrf middleware call handler. The profiler can use line or call profiling. I find line profiling handy when I want to focus into a specific area. While debugging and call profiling require no code changes in your code, line profiling does require you to decorate the function(s) you want to line profile.

The profiler currently requires a patch if you want to do line profiling with Pylons. I’ve sent the patch to the maintainer of line_profiler.

Timer unit: 1e-06 s

File: ...../csrf.py
Function: __call__ at line 40
Total time: 0.004621 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    40                                               @profile
    41                                               def __call__(self, environ, start_response):
    42         1           29     29.0      0.6          request = Request(environ)
    43         1            3      3.0      0.1          session = environ['beaker.session']
    44         1          476    476.0     10.3          csrf_token = session.get('csrf')
    45         1            4      4.0      0.1          if not csrf_token:
    46                                                       csrf_token = session['csrf'] = str(random.getrandbits(128))
    47                                                       session.save()
    48
    49         1            7      7.0      0.2          if request.method == 'POST':

database migrations for SQLAlchemy part duex

Well, as I was looking at making miruku more reliant on sqlalchemy-migrate, I discovered the expirmental command: migrate update_db_from_model! So much for an afternoons work, but at least I’m much more familiar with the migration tools. So here’s how I’ve implemented an auto upgrade for Pylons.

First, easy_install sqlalchemy-migrate

Now, in your pylons development.ini, add the following to app:main:

# SQLAlchemy migration
# if managed, the migration repository is here
migrate.repository = %(here)s/changes
# automatically do database upgrades
migrate.auto = 1

Then, in PRJNAME.config.environment, in load_environment, after the call to init_model add the following:

    # sqlalchemy auto migration
    if asbool(config.get('migrate.auto')):
        try:
            # managed upgrades
            cschema = schema.ControlledSchema.create(engine, config['migrate.repository'])
            cschema.update_db_from_model(meta.Base.metadata)
        except exceptions.InvalidRepositoryError, e:
            # unmanaged upgrades
            diff = schemadiff.getDiffOfModelAgainstDatabase(
                meta.Base.metadata, engine, excludeTables=None)
            genmodel.ModelGenerator(diff).applyModel()

Of course, don’t forget the imports you need:

from paste.deploy.converters import asbool
from migrate.versioning.util import load_model
from migrate.versioning import exceptions, genmodel, schemadiff, schema

Run your app: paster serve –reload development.ini

Now with most basic changes in the model, when paste reloads your database will be updated to reflect the new model. This of course can fail sometimes, such as adding a new column with nullable=False.

I’m only using the unmanaged upgrades right now, so the managed section may need some tweaking, I’ll see when I get there.

database migrations for SQLAlchemy

It was a dark and stormy day, so I skipped stepping outside and worked a while. There’s Vancouver for you.

At Mozilla Messaging we’ve been using Pylons and SQLAlchemy on a couple projects.  One of the features this setup misses that Django and Rails provides is database migration.  Looking around, there’s but one choice for SA since it’s not built in, sqlalchemy-migrate.  Oh, there’s that other project, miruku.  Hmm.

At first glance, miruku seems much simpler, and actually it is a layer on top of migrate-sqlalchemy (sort of).  It doesn’t have upgrade/downgrade versioning, you can only upgrade, but you also don’t have to build up a set of migration scripts.  Just change the table, run the upgrade command, move on.  Sounds much better than figuring out, building and maintaining a set of migration scripts (of course it may not be) at least during development.  So that is where I focused my time.

The first problem I ran into is that it didn’t work with SA declarative classes.  The second problem, despite it being saved from oblivion, there doesn’t seem to be any active maintenance.

Damn, the gauntlet is thrown.  There goes my Sunday afternoon.

The result is a new, working miruku.  I’ve only tested it lightly in a simple Pylons app, and it’s unit tests are specific to using Elixir, but it upgraded my tables for me, even correctly handling a drop column in SQLite (which doesn’t support drop column, miruku does some heavy lifting to make this work).

To use with Pylons, I have to add a section to the ini files (how have ini files survived this long?)
[app:miruku]
sqlalchemy.PRJNAME.url = sqlite://path
miruku.PRJNAME.metadata = PRJNAME.model.meta:Base.metadata

Then setup the miruku support by running miruku:

miruku create --config my.ini --section app:miruku

After that, I can run upgrades with:

miruku upgrade --config my.ini --section app:miruku

There’s a bit more work I plan on doing it, then it may well bit rot again, but here’s my personal wish list for miruku:

  • support altering column properties, miruku only supports add and drop column
  • use more of sqlalchemy-migrate to reduce code size
  • examine table level changes to see if anything major is missing in miruku
  • paster command support
  • better Pylons/Paste integration for configs

If you’re interested in trying it out, drop me a note and let me know how it works for you. You can find miruku in my bitbucket repo.