[Twisted-Python] twisted.web and MySQLdb

[ecn at metaslash.com]

 > Can you qualify "long time"?  And what's the bottleneck?  At the scale 
 > of 100K or so function calls, Python isn't egregiously slow, certainly 
 > not slow enough to not be able to outrun TCP connection setup...?

Hmm, maybe a minute?  I forget exactly.  It was like 4x slower than C++.
(I'm running Linux 2.6test1 with dual AMD Athlon MP/2000's).

I posted a bug in the poll reactor some time ago.  I was testing
an epoll() subclass I had written at the time.

To test the server, I had written a C++ program to connect, 
asynchronously, over the loopback device.  I had to insert a 
pause in the connector to avoid connection failures.  Mind you,
I had to create several "fake" networks in order to have enough
ports for all those simultaneous connections.  This was an absurd
number of connections (10K, 20K, 40K, etc).

The default TCP connection timeout on Linux is ... what? like 61 
seconds or something? That was the number I was hitting.

Everything ran well, up to about 8-10K.  Then it would fall behind,
and then I was at the mercy of TCP retries.

This customer was particularly performance oriented, so I didn't think
that python was going to fly with them, anyhow.  But it is so much
simpler to code/debug/test, I wanted to see how well it would perform.

I tweaked the acceptor to just accept all the connections it could, 
and that was too slow.  It didn't even get into the event loop.  

I also tweaked ACE to accept connections faster, too.  It's 
just a tough problem.

-Eric