[Twisted-Python] twisted.web and MySQLdb

[Rene Dudfield]

Moshe Zadka wrote:

>I won't reply to your question here, I'm sure someone more competent
>than me can do so.
>I will take a minute to note that for some reason, the instinct of
>web developers has been to immediately write applications in the
>LAMP model. While twisted.web is a perfectly good substitute for
>A, and can indeed be used this way, it is much more capable.
>
>LAMP -- Linux Apache MySQL PHP [each of those has variants]
>
>Linux (and most other things in the "L" place, such as FreeBSD) has
>an incredibly fast fork() operation. This leads people to write their
>web code with the process-per-page model. Apache optimizes this by
>preforking, making such code actually scalable enough to deliver good
>response times. However, suddenly there is a problem: since different
>processes are likely to deal with the same user, memory becomes fragile.
>It is unsafe to put things in memory: processes switch users, get born
>and die too often. In Linux, and most other Unices, file locking has
>been a thorny issue. So instead of putting data in files, where the
>subtle semantics of locking and concurrency should be dealt with,
>data tends to end up in the database. That means *all* data. Sessions.
>Temporary "show this message when the user finished his redirect loop".
>Etc. etc. Once all the data is in MySQL, which is the easiest database to
>set up, it becomes natural for the average page to be a "glorified select".
>PHP was designed for this exact scenario: take a select table and spruce
>it up with HTML.
>
>Unfortunately, what LAMP deals to is exactly to this: inevitably, all
>pages are just spruced up select-tables. This makes programming somewhat
>unnatural unless you're programming a database viewer.
>
>With twisted.web, you don't have these problems. It is easy to keep data
>in files, because locking is not an issue. It is easy to cache data in memory
>because everything is served from a single process. If you need to attach
>state to a user's session, you can just keep an object in the session.
>This means programming twisted.web should be a lot more like writing
>a GUI application, and a lot less like writing select-with-HTML.
>You should probably reconsider whether you really want MySQL. It adds
>complexity to your application, and the gain is usually small. Putting
>persistent data in files, and using liberal caching schemes, also plays
>on the core advantage of Linux (and similar) -- it uses the incredibly
>optimized caching algorithms.
>
>
>  
>
RANDOM NOTES ON THIS TOPIC:

Single process has problems too:
- if something blocks or stalls down comes your whole app.  This can 
happen in multi proccess too of course.
- you don't use the operating systems privileges.  You invent your own 
which is often broken and needs to be rewritten a few times.

Of course both of these can happen with your LAMP too.  If the db goes 
down or whatever.  But I guess multiprocess is more resiliant.

Some people find select statements easier.  Often you can do something 
in a line or two which is cumbersum in python.

Berkley db databases are nice to use.  They are quite optimized, and can 
emulate python dictionaries quite nicely.  Then there is zopedb which is 
an option.

Another feature databases have over files is transactions(comming soon 
to a FS near you!).  Making sure stuff is written to files reliably is 
quite difficult.

Databases are accessable from database viewers!  This is handy since 
random python object viewers are not so good :)  You can access 
databases easily from other languages, whereas python objects are harder 
to do.  I guess jelly/xmlrpc/soap helps here if you want to use those.

Databases also handle large volumes of data efficiently(or try to).  
Keeping everything in files is fine as long as you lay it out nicely and 
you have plenty of time to optimize each new query.  However the 
advantages/disadvantages of dbs are discussed to death all over the 
place, and I have nothing new to add.

Be nice to be able to use python objects as memory mapped files.  I 
guess it may be possible.  It has been done with numeric arrays, and 
with c++ objects.  This can be faster than reading files in linux too.

A lot of web pages can be cached into static files which improves 
performance by a lot.  Chucking squid, or apache as proxy in front of 
your normal webserver(s) can be a large help if you are careful.


Have fun!