[Twisted-Python] newbe twisted xmpp load

[Jean-Paul Calderone]

On Tue, 6 Mar 2007 9:44:49 -0500, wrobyn at cox.net wrote:
>Hello,
>I am attempting to write an xmpp load testing thingy in python. I am hitting big walls with regard to creating concurrent connections above 250.. I need really 1-n concurrent connections, the upper limit would be around 500,000 but I would be happy at this point breaching the 250 ceiling.
>Would twisted be able to handle this? Can someone post a simple loop or routine to log in 1-n "people" (i am using the increment in a counter as username) into a jabber server?

>From the ~250 limit, I would guess you're on OS X and running into the
default per-process open file limit.  You can raise this using the `ulimit'
command. If you're on some other platform, then it'd be helpful if you gave
more details about the exact failure you're running into.

Overall, if you use the select reactor, you're limited to FD_SETSIZE, which
varies, but is usually within an order of magnitude of 1000.  If you use the
poll reactor, then you're limited to the number of TCP connections you can
have, which is going to be at most 65000 per IP address (and possibly as low
as 32000).  You might find performance lacking with large numbers of clients
even with the poll reactor, but the epoll reactor should do somewhat better.

I don't know if you'll be able to do 500k with a single process, but I think
you should at least be able to get to the point where the limit you hit is
the CPU, not something related to file descriptors.

Jean-Paul