[Twisted-web] Flamebait

[jonathan vanasco]

Thanks for the input!

I've been working on this one project in my spare time for 3 years now 
-- an online concerts database, cross-referencing cities artists genres 
venues, everything you could imagine, and sending weekly emails based 
on your interests.

I got it  up and running, pretty quickly, in php/mysql 3 years ago -- 
except it was ungodly slow.  Part because of poor design on my fault, 
part because of  the  tools I chose.

Two years  ago, I recoded it into procedural perl and made a lot of 
headway in optimizing it.

One year ago, I recoded it into object oriented perl under mod_perl, 
and had even bigger gains in speed and use.

About the same time though, I started using Python a lot more.  Now, I 
rarely use  anything else.  So I'm trying to code it again, in twisted 
and postgres, for a few reasons:
   1 - i could touch perl a whole lot less, and deal with more 
readable/editable code that i write way faster
   2 - i didn't like the  memory usage of what I had under mod_perl 
(likely  because of my errors, but twisted would be more strict and 
make me keep in line) -- having a single daemon that handles all the 
code just seems smarter and more efficient.

I'd be interested in your approach to caching.  While the bulk of what 
I see this project going is dynamic, certain elements and templates are 
static -- and the constant reading off of disk seems excessive.



On Apr 15, 2004, at 7:51 PM, Stephen Thorne wrote:
>
> I'm going to give you straight answers, because I really don't believe
> in beating around the bush. I could give you curvy twisty statements
> that made you feel better about using twisted instead of apache while
> avoiding all figures, or I could lie to you with statistics by picking
> only the ones that are favourable.
>
> In short, twisted is slower than apache. This is a truth that you (and
> we) should acknowledge. Apache is a very mature web server that has
> stood the test of time. Its not the fastest and its not the best, but
> its pretty darn good.
>
> mod_php and mod_perl are blindingly fast. I can run simple mod_php
> scripts faster than i can serve static files with twisted.
>
> Twisted AND apache are still faster than the average connection. 
> Serving
> static files on a modest 800mhz machine gives me apache being twice as
> fast as twisted, but twisted was able to pump through 12meg/second. If
> you have a connection that exceeds 12meg/second transfers upstream, can
> you please contact me regarding getting an account.
>
> Serving woven was incredibly slow, so much so that with -n 1000 -c 100
> in ab (apachebench) I actually got a timeout. (I was serving a
> DirectoryListing).
>
> So, in face of all this? Should we use twisted?
>
> DEFINATELY. Twisted seriously cuts down on required development time 
> for
> achieving results with certain types of applications, especially
> applications that trancend the web model. Anything stateful, anything
> that wants to integrate with other protocols, anything that has long
> running processing (defer and pickup the result several page views
> later, YES PLEASE!).
>
> Also, a while back i did develop a caching mechanism for woven pages
> that allowed me to increase pageload speed by about 10fold. I think 
> I've
> still got the code laying around somewhere, it was rather simple and
> very effective - based on the existing concept of woven models 
> notifying
> objects of when they change you could invalidate the cache and
> regenerate on the next pageload.
>
> Regards,
> Stephen Thorne.
>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web