[Twisted-web] Performance of twisted web with HTTP/1.1 vs. HTTP/1.0

[Jason E. Sibre]

***  Probably not a problem with Twisted...  see my last comments ***

> Well, any open source tester, even a simple dodgy one you write
> yourself --
> I don't have any strong preference, but it would be nice to be able to
> reproduce your results :)

Well, I've attempted to write a script in python to do this, but, I'm not
seeing the performance difference when I run it (It performs well, whether
it's 1.0 or 1.1.  Actually, it performs slightly BETTER if it's 1.1, but
only about 10%)!  After rubbing my eyes, and trying again (and again, and
again....)  I've concluded that I'm NOT losing my mind (the perf degradation
is very real and noticable when loading/refreshing a page with lot's of
small images), but for some reason, the way httplib constructs an HTTP/1.1
request doesn't trigger it (while ACT, Internet Explorer, and Mozilla 1.7b
do trigger it)

> I can't think of what else to suggest, except for running the
> server in the
> Python profiler (the -p=profile.log switch to twistd, if you're using it),
> and seeing if that reveals where the extra time is being spent.

Hmm...  I'm not running twistd, but maybe I can do something along those
lines in my script that kicks Twisted off...  It already has some profiling
bootstrap code in it.  I hate wading through all the profiler output, but at
least this may be easy to spot, since it's so dramatic.

> Maybe you could insert a "print repr(line)" into HTTPChannel's
> lineReceiver
> handler (see twisted/protocols/http.py) to get a more raw dump of the
> request, and double-check that there's no significant differences in the
> HTTP/1.0 vs. HTTP/1.1 requests, but I'm doubtful...

Another good suggestion.  I don't know where that is at the moment, but I'm
sure it'll be easy to find once I dig/grep for it.  I assume that's a pretty
low level spot in twisted's comm stack?

> Also, to eliminate another difference between your environment
> and mine, is
> it possible to run your tester against a Twisted server on a
> linux box, and
> verify that it still gives the same behaviour?  If for some bizarre reason
> it turns out to be a windows-only issue, it'd be good to know
> before I waste
> too much time trying to reproduce it on Linux :)

Actually, I am running it on a Linux box.  Of course I ran ACT from a
windows box, but they're sitting next to each other on a 100mbps lan.  The
data transfer rate was not a factor (Apache on Linux can hand about 500
images / second to the windows box).  I'm using Python 2.3 on SuSE 8.1 with
a 2.4.24 kernel (I compiled it myself, but I haven't had any troubles with
it -- that I know of ;) )


> Also, I'm losing track of what you have and haven't tried... it might be
> time we opened a bug report for this, so we can track this properly.

I'd be up for that, but I'd really like to see at least one other person
duplicate it before we went that far.  I may just have something stupid
going on over here.
----------------------------------------------------------------
Ok, you know what?  I'm starting to think it's a not a problem intrinsic
with twisted.  I just put together a html page with 100 small images, and
served it through the Qx / Twisted combo and it was slow.  Through Qx /
Medusa, it was fast.  Through Twisted by itself, it was fast (using Moz
1.7b).  Here's the script I used:

#!/usr/bin/python
from twisted.internet import reactor
from twisted.web import static, server

root = static.File(
  "/home/jsibre/programs/python/sitehostApp/sitehost/web/static/test")
reactor.listenTCP(8082, server.Site(root))
reactor.run()

It ran really fast, all images loaded in about a second (through the Qx /
Twisted combo, you can watch them load one after the other...  Takes about 8
or 9 seconds).

I was afraid of this... It's probably something in the glue that ties the Qx
to Twisted.  I'll work it from that angle for a while and see if I find a
problem that is fixable, and provide the fix to the Qx folks when I figure
it out.

Thanks for your help, Andrew.

Jason