[Twisted-Python] Twisted's ability to handle many open connections.

[Terry Jones]

>>>>> "glyph" == glyph  <glyph at divmod.com> writes:

glyph> That's more or less accurate.  It sounds like you understand this,
glyph> but because I can never repeat it enough: Deferreds don't do
glyph> anything magic or get involved with select() directly

Ok, got it.

glyph> On Windows, there is a reactor that supports a maximum of 63
glyph> concurrent events (assuming your process has no windows open).  On
glyph> UNIX the default reactor is still select() based and will definitely
glyph> bog down if you have too many simultaneous sockets open.  Sometimes
glyph> it is handled quite badly!

glyph> However, the selection of a reactor is a configuration option, not
glyph> an application-architecture issue (unless, of course, you are
glyph> writing a GUI which has requirements for what event loop it will run
glyph> on).

glyph> You can choose a reactor with the -r argument to twistd, trial, and
glyph> other similar tools like axiomatic.  You can choose different
glyph> reactors for different platforms.  Application code should be none
glyph> the wiser.

Thanks for the details. Sounds like I can handle the problem, if it ever
happens, by simply bumping FD_SETSIZE and rebuilding the kernel, then
proceeding as normal. And/or drop old connections.

glyph> If your application has such a case, though, implementing an epoll,
glyph> aio, kqueue, libevent, or <insert your totally non-portable,
glyph> OS-specific, crazily scalable IO multiplexing API here> reactor
glyph> should not be an inordinate amount of work.  Less work, at least,
glyph> than building such a beast and then writing what amounts to 2/3 of
glyph> the Twisted core around it anyway :).

Ok, great. I'm actually backing my way out of using lots of already written
low-level asynchronous networking code, my own RPC mechanism, etc., in
favor of letting someone else do that work :-) So I do know what you mean,
and agree.

Thanks again.

Terry