Archive for the 'Python' Category

using json-schema for exploring api servers

A while ago Google introduced an API Explorer for many of their APIs.  In Mozilla Messaging we had done a similar api browser for Raindrop that James Burke created, which we ended up using in the early days of F1 as well.  I’ve been meaning to make our api browser generic so that it could be used with any api server, and had played with it off and on over the past couple months.  In the last couple weeks I’ve found a few hours here and there to get that idea to a working state, at least for Python.  Here is an example of the API browser, along with a response from testing the api (lower part of right panel).

Following Googles information on API Discovery, I used json-schema for the base data format.  I then hacked up a UI from another project (design by Andy Chung) to show documentation based on that schema, and to allow interacting with the API through a form.  Since it’s based on json-schema, it can at least display the documentation for any API that provides a json-schema file.

Another big benefit of using json-schema is that you should be able to use the generic API clients that google has created against your own API servers, which only lacks in a pure Javascript implementation (they use GWT).

Here’s the same frontend showing the Google Books API:

In the server side Python code, I use a set of decorators for defining the top level application documentation and the documentation for each api endpoint.  Information from Routes is used to find the api documentation, then the json-schema is put together and cached on the first call to the server. A cool benefit is that I also use the schema decorators to validate api input against the json-schema.  You can see my full implementation in the apibase project on github.

Provides:

  • decorators for Paste/Routes based applications
  • input validation for API endpoints
  • controller for retrieving schema
  • simple browser to the API
  • ability to use generic clients, no more writing client side api modules

Missing, or what might be next when I have time:

  • Authentication controls for protected APIs (e.g. Googles Explorer handles client side oauth)
  • Generally needs work and cleanup
  • Support for more than one API at a time in the browser, perhaps support remote servers that support CORS
  • Support for parameter constraints in the API browsers test form (e.g. min/max values for integers), although validation of those should be working in the server
  • Possibly use more information from Routes in generating json-schema
  • Expose more documentation in the browser (e.g. HTTP method of call)
  • Create a pure Javascript client library to work against json-schema enabled APIs
  • Servers for other languages? (e.g. node.js)
  • other ?

 

database migrations for SQLAlchemy

It was a dark and stormy day, so I skipped stepping outside and worked a while. There’s Vancouver for you.

At Mozilla Messaging we’ve been using Pylons and SQLAlchemy on a couple projects.  One of the features this setup misses that Django and Rails provides is database migration.  Looking around, there’s but one choice for SA since it’s not built in, sqlalchemy-migrate.  Oh, there’s that other project, miruku.  Hmm.

At first glance, miruku seems much simpler, and actually it is a layer on top of migrate-sqlalchemy (sort of).  It doesn’t have upgrade/downgrade versioning, you can only upgrade, but you also don’t have to build up a set of migration scripts.  Just change the table, run the upgrade command, move on.  Sounds much better than figuring out, building and maintaining a set of migration scripts (of course it may not be) at least during development.  So that is where I focused my time.

The first problem I ran into is that it didn’t work with SA declarative classes.  The second problem, despite it being saved from oblivion, there doesn’t seem to be any active maintenance.

Damn, the gauntlet is thrown.  There goes my Sunday afternoon.

The result is a new, working miruku.  I’ve only tested it lightly in a simple Pylons app, and it’s unit tests are specific to using Elixir, but it upgraded my tables for me, even correctly handling a drop column in SQLite (which doesn’t support drop column, miruku does some heavy lifting to make this work).

To use with Pylons, I have to add a section to the ini files (how have ini files survived this long?)
[app:miruku]
sqlalchemy.PRJNAME.url = sqlite://path
miruku.PRJNAME.metadata = PRJNAME.model.meta:Base.metadata

Then setup the miruku support by running miruku:

miruku create --config my.ini --section app:miruku

After that, I can run upgrades with:

miruku upgrade --config my.ini --section app:miruku

There’s a bit more work I plan on doing it, then it may well bit rot again, but here’s my personal wish list for miruku:

  • support altering column properties, miruku only supports add and drop column
  • use more of sqlalchemy-migrate to reduce code size
  • examine table level changes to see if anything major is missing in miruku
  • paster command support
  • better Pylons/Paste integration for configs

If you’re interested in trying it out, drop me a note and let me know how it works for you. You can find miruku in my bitbucket repo.