[Twisted-Python] Twisted Performance

[Laurens Van Houtven]

On Wed, Oct 14, 2009 at 4:44 AM, Daniel Griffin <dgriff1 at gmail.com> wrote:
> Hi,
>
> I am new to twisted and have been having trouble finding out information
> about twisted's performance. I have a fairly simple setup where I need to
> open a bunch of TCP connections that last for varying amounts of time but
> dont do much. I have tried using threads(got GILed to death) and
> Processes(even worse). Now I am looking at either making a system to start
> the connection and send info to have the remote point "phone home" when its
> done, then closing the connection or using something like Twisted.
>
> my socket conversation:
> my app -> send a message that triggers an action on the other end
> other end -> recv's message does action(can take any amount of time)
> other end - > sends results back to my app

Sure, that's a blueprint for pretty much every Twisted app. You just
need to make the thing that takes a long time not block the reactor.
How you do that mostly depends on what the "long thing" is.

Usually this means using an existing library, sometimes it means
writing your own, and for some unfortunate cases it means deferring to
a thread. (Note that deferring to a thread to fake non-blocking IO is
not anywhere near as bad as doing actual _work_ in threads ;-))

Personally I'm a big fan of this series of blog posts:
http://krondo.com/blog/?page_id=1327

And you should probably read the finger tutorial and the deferreds tutorial :-)

> Can twisted handle up to several hundred connections like this? Is there a
> better approach? Is there anything I should avoid?

Depends how much work you're doing in the Twisted process or on the
same box, of course. You're probably looking at several thousand
rather than several hundred, depending on how much work needs to be
done in each one.

> Thanks,
>
> Dan