[Twisted-Python] Configuration, Persistance and Version Control

[Glyph Lefkowitz]

Kickin' it IRC style --

Here are some responses interspersed with the dialogue.  Hope this helps
:)

On Sat, 28 Jul 2001, Kevin Turner wrote:

> <itamar|home> you coifgure them and then can pickle them to disk

... OR you run them, depending on your requirements.  The default 'twistd'
binary does UNIX daemonization; but if you wanted to do an NT service,
you'd obviously have to have some different code.

> <Tv> Acapnotic: But all that does is make your "config settings"
>           into "command line settings".

It makes your *initial* "config settings" into "command line settings".  
Those scripts exist more for examples than for actual configuration; the
"settings" they provide are a small subset of the full set available in
the code.

> <Tv> I don't see it making things very easy.
> <Tv> And think about version upgrades.

We here at Twisted Matrix Laboratories have a commitment to backwards
compatibility :-).  *IF* we had more users, the 0.9.0 upgrade would have
gone much more smoothly; we actually had a backwards compatibility plan
and decided to forego it for efficiency because we (rightly) believed that
nobody would care.

My configuration (e.g. the www.twistedmatrix.com site) is fairly complex,
and I've been maintaining it with Twisted since the inception of
Twisted.Web (which I think was .7 or so).  It's never taken me more than 3
or 4 minutes to replicate this with simple python scripts.

> <Tv> I see trouble with the approach; mostly losing the unix
>           ubiquitousness (sp?). I can grep my config files; I store them in
>           CVS.

1: learn how to use add +kb :-)

2: use pickle's "text" mode.

3: version control a script which generates and runs your monumentally
byzantine configuration, then store THAT in cvs.

About option 3: radix, moshez, and I have been thinking about, and
prototyping configuration systems for quite a while.  I have yet to think
of a system which can't be easily beaten by "just using Python".  For
example, here's an example XML solution;

<Site name="myvhost">
 <NameVirtualHost>
  <Host name="myhost.com">
   <Resource>
    ...
   </Resource>
  </Host>
 </NameVirtualHost>
</Site>
...
<WebServer port=80 site="myvhost"/>

And here's the actual Twisted code which will do the same thing (I
haven't had an opportunity to test it, but you get the idea):

---
from twisted.web.server import Site
from twisted.web.vhost import NameVirtualHost
from twisted.internet.main impot Application
a = Application("web")
v = NameVirtualHost()
r = SomeResource(...)
v.addHost("myhost.com", r)
s = Site()
s.resource = v
a.listenOn(80, s)
---

It's equally concise *and* more flexible.

> <Tv> And don't give a trivial example..
> <Tv> I have multi-hundred-line config files here.
> <Tv> Shared between n hosts, via CVS.
> <Tv> I'd love to hear more of the twisted philosophy..

If you have multi-hundred-line configuration files some of which allow
programmability), I'm guessing that you have considerable applications
that are written in buggy half-thought-out programming languages with no
module systems and you're using version control as part of a crutch to
make up for their many and varied deficiencies.

Since Twisted will allow you to use Python (or save your configuration in
Pickles, which is usually more convenient -- keep in mind I use this
software myself; pickles are how I use it -- to configure your servers,
you can encapsulate parts of your configuration or segment it however you
like.  and VC it too.

> <Tv> Personally, I have _needed_ programmability in config files
>           many times.

And you've got it ;)

> <Tv> Acapnotic: Oh, I like version control. I wish I could put more
>           things into it.

Hopefully jasbahr will jump in here with a little comment about potential
future ideas for version control within Twisted itself... ;-)

> <Tv> Have you ever screwed up a config file?
> <Tv> Have you ever downgraded something, and found out it won't
>           understand the new config file, and had to recreate the config to
>           match the older version?
> <Tv> The same reasons you version control source code.

> <Acapnotic> tv: you should post to the list at
>           twisted-python at twistedmatrix.com
> <Tv> Acapnotic: I'm not following any more lists until I have
>           migrated my mail to another domain :)
> <Acapnotic> tv: bah, you don't have to follow it, just post ;)

I would *really* appreciate it if more people took advantage of resources
like the mailing list, and my personal email address :-), to resolve
issues like this.  Thanks, kevin, for posting this log!

> <Acapnotic> Tv: so if you want programmable config, is there a
>           problem with just using python?  In which case your configuration
>           process is a script that does "app = pickle.load(...) app['web'].
>           port = 8080 ; app.append(ChickenServer()) ; ... ; pickle.dump(app, 
>           ...) ?
> <Tv> Acapnotic: That may very well be what I'd do. But then the
>           mechanism needs to support _both_ getting config programmatically
>           and state from pickle, at the same time.

Just make your "configuration file" as described, except put "app.run()"
at the end of it.  This lets you do what "configuration" really is; e.g.
to make a customized version of the application for your site.

> <Acapnotic> Well, if I follow this train, there is only the state
>           of the application, since it is persistant.  That state may be
>           changed or examined programmaticly.

It's *optionally* persistent.

The point of Twisted's configuration philosophy is that "configuration" is
not really a disjoint set of options, but rather the custom development of
a personal application.  Twisted provides you with a big pile of very
well-developed pieces (so you hopefully don't have to spend time fixing
the little pieces while you're building your big one), but you still have
to put them together somehow.  The mk*server scripts provide one way to
do that; hopefully matsaleh's upcoming configuration protoserver will
allow it to be done interactively so that users can discover more aspects
of the software.

In eschewing the ubiquity of "configuration files" on UNIX, twisted also
eschews the necessity of dedicating a large body of code to parsing,
interpreting, and understanding an underpowered mini-language designed
only for talking to it.  "Configure" with python.  "Script" with python.  
"Code" with python.  "Debug" with python.

The twisted philosophy is that all of these tasks are inherently the same,
and are all complex enough to warrant a full-powered programming language.  
Event-based dynamic modification of a stateful system is what object
orientation was originally intended to do.

> <Tv> I see something beautiful there, but..
> <Tv> noone has yet explained persistence to me well enough to
>           convince me.
> <Acapnotic> but what happens when you upgrade?  heh.
> <Tv> What if a buggy program corrupts its state?
> <Tv> Then you lose your config file also.

This assumes you don't back up your original config file.  Why would you
throw it away?  If it were complex, I'd keep mine in CVS :)

> <Tv> There needs to be something else.
> <Tv> Something that is less volatile.
> <DavidC_> what do you mean? both config and state are the same
>           thing.
> <Tv> No.
> <Tv> config changes when *I* say it changes.

Config is *initial* state.  e.g. the state that you start the server up
with; so it changes whenever the program changes.  Or does anything.  You
can always stop the server and start from scratch again, but that doesn't
mean that bugs won't affect you.

> <itamar|home> nah, config is parsed and then "rendered" into state
> <Tv> program can change it's state as it wants.
> <Tv> Any bug in program must never change the port to bind to.

If the bug would change the persistent port, then it would change the
runtime port.  That seems like it would be pretty bad too :)

> <Tv> :)
> <Tv> I see config as sort of a factory for initial program states,
>           if you want to put it that way.

If one were to believe there is something beyond this that describes
"configuration", then they would be confused :-).  The *state* of the
server changes whenever it executes an action that causes state to change;
the difference between Twisted and other systems is that Twisted *can*
remember these changes to state automatically.  This state will not
replace your initial state, nor will it prevent you from writing scripts
which create the initial state each time that the server is run.

The reason I suggest the pickle-the-state approach is because if you make
a change to the "configuration" as the server is running, you can change
it *live*, test it, etc, and know that when the server comes up next time,
it'll start the same way, rather than having to change your "factory",
shut down your server and start it again, or make the change to your
factory and your runtime state concurrently and take the risk that they'll
be different.

> <itamar|home> lets say by mistake someone does "del serverInstace.
>           port"
> <itamar|home> and then shuts down the server
> <itamar|home> the pickled state is now wrong

The pickled state that the server saved *when it shut down* is now wrong.  
The initial state (what *I* consider the "configuration") is still
correct; but you now have the corrupt state to study as well (so you can
figure out what happened with information beyond the log.)

> <DavidC_> well that would be a bug.
> <itamar|home> the point is that if you have an external, version
>           controleld config file
> <itamar|home> these bugs don't bite you, or even if they do at
>           least they don't effect the configuration

In conclusion, "so do that then" :).

I still don't understand how a pickled file isn't "external".  It's not
like it gets mmap'd when you start the Python process and then
automatically overwritten after every action that happens :)

                      ______      __   __  _____  _     _
                     |  ____ |      \_/   |_____] |_____|
                     |_____| |_____  |    |       |     |
                     @ t w i s t e d m a t r i x  . c o m
                     http://twistedmatrix.com/users/glyph