[Twisted-Python] Twisted webserver performance issues (was volunteer...)

[exarkun at twistedmatrix.com]

On 10:41 pm, leyssw at ihug.co.nz wrote:
>
>Isn't the simplest option to place a decent reverse proxy between the 
>webserver and our clients?

Possibly so!  I gave this a half-hearted attempted a few weeks ago but 
quickly gave up.  If someone who is familiar with configuring such a 
proxy would like to help out, perhaps we can get something useful up and 
running.
>
>For example, varnish will cache and proxy simple GETs, so simple views 
>of the front page and docs (which I imagine is a large chunk of the 
>traffic) will be offloaded from the webserver:
>http://varnish.projects.linpro.no/wiki/VarnishFeatures
>
>Also, is the current installation (Db, apache + trac) all on one 
>machine or are the DB and the webserver split already?

They're all on one machine.  Changing this is an expensive proposition 
with a hard to predict payoff, so it's rather low on the list of ideas 
to try.
>These would be easy ways to throw more hardware at the problem.
>
>What is the current web-server configuration?

Can you be more specific about which configuration you're interested in?
>Can you share traffic stats for the top 20 pages, so we get an idea of 
>where the load is?

Yep.  Here are counts for the last few days.  Rank, request count, URL.

1)      22058   /trac/
2)      8742 
/trac/attachment/wiki/TSF/SponsorLogos/seomoves.png?format=raw
3)      8646 
/trac/attachment/wiki/TSF/SponsorLogos/zenoss.2.png?format=raw
4)      8550    /trac/attachment/wiki/TSF/SponsorLogos/Appropriate- 
Solutions-Inc.gif?format=raw
5)      8499 
/trac/attachment/wiki/TSF/SponsorLogos/securitymetrics.png?format=raw
6)      3075    /trac/wiki/Documentation
7)      2615 /trac/timeline?changeset=on&max=100&daysback=10&format=rss
8)      2448    /trac/wiki/Downloads
9)      1788    /trac/wiki/FrequentlyAskedQuestions
10)     1356    /trac/wiki
11)     1019    /trac/wiki/TwistedProject
12)     966     /trac/wiki/TracGuide
13)     828     /trac/wiki/TwistedWeb
14)     814     /trac/robots.txt
15)     781     /trac/search
16)     710     /trac/wiki/TwistedProjects
17)     662     /trac/wiki/WikiStart?format=txt
18)     605     /trac/wiki/TwistedAdvantage
19)     595     /trac/report/9
20)     590 
/trac/timeline?milestone=on&ticket=on&changeset=on&wiki=on&max=50&daysback=90&format=rss

Looking at this, I'm reminded that we should try moving the sponsor 
logos to somewhere else

Unfortunately one thing we don't know is the relative cost of serving 
each of these URLs.  The sponsor logos, while accounting for a 
significant number requests, may actually be a very small contribution 
to overall load.

Anyhow, thanks!

Jean-Paul