[Twisted-Python] Re: Getting Stories Straight (RPYs)

[Glyph Lefkowitz]

On Wednesday, June 25, 2003, at 09:40 AM, W.J. wrote:

>>> You don't want to bounce offline all users currently using the IMAP 
>>> and
>>> IRC servers every time you make a small change to a web page.
>>
>> If you're doing development on the live server the users are using,
>> bouncing them off every now and then is a good way to let them know
>> they are morons for using such a sketchy server.
>
> Very much in the same way that moving a chair or changing a lightbulb
> is reason enough for kicking everyone out of the building.

I think there is a slight miscommunication in terms of what is meant by 
'every time you make a small change'.  I get the impression that what 
is being said here is:

	<moshez> you should not be doing your development on a live server.
	<W.J.> so you're saying I shouldn't be able to load updates on a live 
server?

One of Twisted's original design goals was _fast server restart_, 
because restarting the server is pretty much inevitable if you want to 
make substantial changes; I wanted to make it super-easy for developers 
to bounce their servers every minute while making minor changes.  I 
even had a 'bounce my Twisted server' key in emacs for a while when I 
was working on a more deeply interactive system than I am now.  This 
was based on some pretty bitter experience with a server that I was 
coding on at the time which took almost an hour to start up.  EJB 
servers also have this problem; it's not just that you have to restart 
them, it's that you have to _wait_ when they restart.

Another design goal is _restart-free code reload_, which you can see 
still in twisted.python.rebuild, because I know there are times when a 
deployed server needs to be fixed _right now_ and although it can lose 
a few seconds of time, it could not brook an interruption of service.

The choice of how you do your development with these tools is really a 
matter of taste, and I don't have any feelings about it one way or 
another; however, deployment is contingent upon how reliable restart 
vs. reload is.

You should NEVER make live server code changes without first setting up 
a test environment, making the changes, testing the changes, and then 
bringing up an OLD server environment and testing the upgrade path from 
one version to the other however you intend to do it live.

Python's reloading is handy, but dangerous, and Twisted's 'rebuild' 
utility adds to both the power and the danger.  If you use Python's 
reload() thinking it will simply update your server to the most recent 
code revision, you will be in for a nasty shock when you find that 
objects using both the new AND the old code are still around, 
isinstance() will fail sometimes for instances of a particular class, 
and tracebacks have incorrect code in them.

rebuild() 'does more' for you, however, it can still be surprising.  
You may find that objects which were previously initialized to a 
consistent state are now missing attributes which the new code expects 
the initializer to set.  You may find that objects that were old-style 
and are now new-style simply didn't reload.  It has been too long since 
I've looked at the code to remember every dark corner, but there are 
definitely things that can be made to fail.  You have to be careful 
with what kinds of changes you load, and understand how the change will 
interact with the existing code and objects you're updating.

So, despite the conveniences that Twisted can provide for you, you 
can't escape the inevitable software reality that "Development", 
"Test", and "Production" should be three completely separate 
installations on which you can perform upgrades independently, in a way 
suitable to their purpose.  Anything less will result in an unstable 
production environment where, whether you like it or not, your users 
will all fall out of a 30th story window every time you change a 
certain lightbulb.