[Twisted-Python] twisted.persisted.journal: a prevayler-inspired persistence system

[Itamar Shtull-Trauring]

I've just checked in Journal, a persistence system inspired by Prevayler (http://www.prevayler.org for details). The idea is basically that all interactions with the system are done via serializable commands objects, which are logged *before* they execute. This allows us to keep down number of transactions in the system since in the worst case, the system can be restored by running the logged commands.

Additional benefits include being able to rerun sessions for testing performances, changing the internal data model is easy (just delete snapshot of system state and rerun all commands against new codebase), and so on.

This is an alternative to the immutable database model of keeping old versions of data - instead, we keep a log of all state changing commands (or even queries - useful for cases when you need to keep track of every read to the system).

The main difference from prevayler is flexibility - while I do support the "everything in memory" model, I also have code for using RDBMS for storage as well, using twisted.enterprise.row. In the future journal will support logging commands to a RDBMS as well, allowing you to easily gather statistics about everything that has ever been done to your system. The RDBMS code is untested.

Plugging the code into any persistence mechanism for snapshotting purposes should be possible, e.g. you could use ZODB as well. However some will be more suited than others - t.e.row is pretty good since it doesn't require a transaction for each command executed.

Example of using the system (with memory snapshotting and pickle based log):
http://www.twistedmatrix.com/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/twisted/test/test_journal.py?rev=1.1&content-type=text/vnd.viewcvs-markup&cvsroot=Twisted

-- 
Itamar Shtull-Trauring    http://itamarst.org/
Available for Python, Twisted, Zope and Java consulting