[Twisted-Python] Configuration, Persistance and Version Control

[Kevin Turner]

So, Tv and itamar were on channel today, asking about Twisted
philosophy, persistance, and configurability.  Lacking the
presence of anyone closer to the pure Twisted light, I tried
to be a good little minion and make the case...  But since
I'm the guy who doesn't quite believe in .spl files, I wasn't
able to be terribly convincing.

And now I could try to play his side of the argument with you guys,
but ya know, it'd be so much easier for me to just let his words do
the talking.  Scrollback excerpts:

<itamar|home> anyone who knows how twisted's config mechanism works?
<itamar|home> typically, glyph is on 24 hours a day unless I
          actually want to talk to a twisted developer
<Acapnotic> Wait, Twisted has a config mechanism?
<itamar|home> well, its server are persistent
<Acapnotic> right
<itamar|home> you coifgure them and then can pickle them to disk
<Acapnotic> right
<itamar|home> my question is: how do I configure them?
<itamar|home> can this mechanism be used with other software
          packages?
<Acapnotic> You load them up, set them, and re-pickle them.  You
          can also set them through the interactive python interface on the
          telnet server, say
<Acapnotic> Now somehow, Glyph claims that this is easier than
          having configuration files.
<Acapnotic> I am not yet convinced of that, however.
<itamar|home> I am
<itamar|home> my software right now needs either something like
          that or an XML config file
<itamar|home> which'll end up being a pickle equivalent anyway, so
          there's no point

[...]

<Acapnotic> itamar: Well, so far as I've been able to tell, there's
          not overmuch magic there.  'mkfooserver' just creates an instance
          of a server object and pickle.dumps it, twistd just pickle.loads it
          and invokes its the run() method.
<itamar|home> Acapnotic: well, my configs are pretty complex
<Tv> Acapnotic: But all that does is make your "config settings"
          into "command line settings".
<Tv> I don't see it making things very easy.
<Tv> And think about version upgrades.

[...]

<Tv> I see trouble with the approach; mostly losing the unix
          ubiquitousness (sp?). I can grep my config files; I store them in
          CVS.
<Tv> DavidC_: You want every program to contain some UI or command
          mechanism to fully configure it.
<Tv> There's a lot of things I'd wish someone would spell out about
          twisted. One thing is "how do I version control my configuration?"
<Tv> And don't give a trivial example..
<Tv> I have multi-hundred-line config files here.
<Tv> Shared between n hosts, via CVS.
<Tv> I'd love to hear more of the twisted philosophy..

[...]

<Tv> Personally, I have _needed_ programmability in config files
          many times.

[...]

<Tv> Acapnotic: Oh, I like version control. I wish I could put more
          things into it.
<Tv> Have you ever screwed up a config file?
<Tv> Have you ever downgraded something, and found out it won't
          understand the new config file, and had to recreate the config to
          match the older version?
<Tv> The same reasons you version control source code.

<Acapnotic> tv: you should post to the list at
          twisted-python at twistedmatrix.com
<Tv> Acapnotic: I'm not following any more lists until I have
          migrated my mail to another domain :)
<Acapnotic> tv: bah, you don't have to follow it, just post ;)

[...]

<Acapnotic> Tv: so if you want programmable config, is there a
          problem with just using python?  In which case your configuration
          process is a script that does "app = pickle.load(...) app['web'].
          port = 8080 ; app.append(ChickenServer()) ; ... ; pickle.dump(app, 
          ...) ?
<Tv> Acapnotic: That may very well be what I'd do. But then the
          mechanism needs to support _both_ getting config programmatically
          and state from pickle, at the same time.
<Acapnotic> Well, if I follow this train, there is only the state
          of the application, since it is persistant.  That state may be
          changed or examined programmaticly.
<Tv> I see something beautiful there, but..
<Tv> noone has yet explained persistence to me well enough to
          convince me.
<Acapnotic> but what happens when you upgrade?  heh.
<Tv> What if a buggy program corrupts its state?
<Tv> Then you lose your config file also.
<Tv> There needs to be something else.
<Tv> Something that is less volatile.
<DavidC_> what do you mean? both config and state are the same
          thing.
<Tv> No.
<Tv> config changes when *I* say it changes.
<itamar|home> nah, config is parsed and then "rendered" into state
<Tv> program can change it's state as it wants.
<Tv> Any bug in program must never change the port to bind to.
<Tv> :)
<Tv> I see config as sort of a factory for initial program states,
          if you want to put it that way.
<itamar|home> lets say by mistake someone does "del serverInstace.
          port"
<itamar|home> and then shuts down the server
<itamar|home> the pickled state is now wrong
<DavidC_> well that would be a bug.
<itamar|home> the point is that if you have an external, version
          controleld config file
<itamar|home> these bugs don't bite you, or even if they do at
          least they don't effect the configuration