[Twisted-Python] .tap and twistd and persistence

[glyph at divmod.com]

On Mon, 17 Apr 2006 12:34:56 -0400, Matt Feifarek <matt.feifarek+twisted at gmail.com> wrote:

>So, if the docs tell us that this way is the way to use twisted, and they're
>wrong, where do we go to learn how to use twisted?

Apparently you've come to the right place. :-)

>The docs are quite clear about using application objects and making your
>code into a twisted 'plugin' rather than simply poking at the reactor (as in
>some of the quick-n-dirty tutorials).

I have to apologize for that; .tap plugins, in particular, are my fault.  Sometimes, a system will be written, documented, people will use it for a while, and it will turn out that the supposed benefits of putting up with the pain of a plugin interface or a layer of indirection are actually not worthwhile.  There is chaos for a while as everyone tries to figure out what the best way to solve the problem that the interface originally wanted to solve is.  I wish Twisted didn't have this problem, but it's hardly unique.  The difference is, we admit we have the problem, let our users know we are reconsidering our solution, and attempt to discover the correct way :-).

(For a counterexample, see the web stuff in J2EE.  The XML addicts who designed it refused to admit they were wrong or that the user community was having trouble with the mountains of configuration required, and now the "right" way to use J2EE for Web seems to be ditching it in favor of Ruby on Rails...)

Right now the appropriate thing to do is use a .tac file.  Some of what the documentation says is in fact correct: you still shouldn't poke at the reactor directly: just create the Application object (as documented in those howtos) in a .tac configuration file rather than an axiom plugin.

One of the major enhancements we need to make (this will probably be seen in a Twisted 3.0 release) is to make a publicly-supported version of twisted.scripts.twistd.ServerOptions.  If you are uncomfortable reading grotty platform-specific code and getting into spirited arguments on this mailing list about broken compatibility from release to release, that is not yet the class for you.  Eventually we will provide a nice way to make a twistd-like command-line program (and hopefully also a similar GUI-friendly equivalent) that adds additional arguments and startup logic of your choosing.  However, some projects written by Twisted core team members are already abusing the private ServerOptions interface to provide this same effect.

One such project is something that I wrote, Divmod Axiom ( http://divmod.org/trac/wiki/DivmodAxiom ).  Axiom abuses this internal interface to implement a rather nice mechanism that actually accomplishes what .tap plugins set out to do.  Axiom is an object database with native support for the Twisted programming model.  The feature I am talking about specifically here though, is a nifty command-line tool called 'axiomatic'.  Axiomatic provides a plugin architecture, like tap plugins, which perform manipulations on a database, and a "start" sub-command, which allows you to do things like this:

     % axiomatic -d mymantissa.axiom mantissa
     Enter Divmod™ Mantissa™ password for 'admin at localhost':
     Confirm Divmod™ Mantissa™ password for 'admin at localhost':
     % axiomatic -d mymantissa.axiom start -n

The first command would be roughly equivalent to a hypothetical mktap plugin:

    % mktap -a mymantissa.tap mantissa
    % twistd -nf mymantissa.tap

The difference is that Axiom is a real object database, based around SQLite, with support for managed transactions, schema evolution, persistent timed events, and more.  You can therefore put a reasonable amount of real data into it in addition to your configuration.

>I can certainly write my own persistence code, but I'm a little worried that
>more of the "best practices" presented by the docs are wrong. Should I still
>use twistd? Should I use the application object? Etc.

Yes, use twistd.  Yes, use the application object.  You *can* even write mktap plugins; they will work as described by the documentation.  They're not buggy.  (Okay, they're not *very* buggy.)  It's just that the advertised benefits are not really so great so most people don't use them any more.  The ones that come included with Twisted are handy toys.

A separate issue is that if you have large, complex configurations, storing them in pickle (or any automatic persistence mechanism) is dangerous.  A chance upgrade of any Twisted module which your configuration happens to contain instances from, or the standard library, or any other library which Twisted depends on, or any of your code... well, basically if any Python code on your computer changes at all, you could end up with a configuration that won't load.

This isn't really a Twisted-specific issue, but it is a lesson that many Twisted Matrix Labs developers have learned by way of .tap files.  There is a well-written paper on the dangers of automatic persistence here:

   http://www.erights.org/data/serial/jhu-paper/upgrade.html

If you're intrigued by Axiom after reading this, there's good news and bad news.

The good news:  Axiom can provide all the advantages that tap plugins were supposed to, along with a relatively scalable and powerful general persistence mechanism.  The dangers of upgrading are far reduced from .tap files, because (A) axiom includes very good schema migration support and automatic upgrades between released versions, (B) all attributes are explicit, so there are no "hidden surprises" when you upgrade, and (C) only Item objects in the database need to be upgraded, not every random thing you invoke in your code.

The bad news:  Axiom is heavily in development, nowhere near as stable as Twisted.  The 0.4 release is ancient, in terms of available features, and documentation is mainly being written for the main line of development.  Unfortunately there aren't necessarily upgrades beween every random revision of SVN trunk.  In practice there have been relatively few breakages.

However, it sounds like your needs are relatively lightweight and that you could happily use the 0.4 release of Axiom to deploy a real application.  When we do get around to producing a new release, you shouldn't have to worry too much about backwards compatibility.