[Twisted-Python] shared list in factory, mutexes?
Gabriel Rossetti
mailing_lists at evotex.ch
Tue Apr 15 03:55:49 MDT 2008
Andrew Bennetts wrote:
> Gabriel Rossetti wrote:
> [...]
>
>> think what I don't get is that with thread network code, each thread is
>> an individual self-contained unit that does whatever processing it needs
>> to do however long it may need to take, and the OS takes care of
>> scheduling each thread, but with twisted, the main thread executes each
>> connection one by one sequentially and thus if one of them needs to to
>> some lengthy processing, it will block all other connections from being
>> processed, so you have to use threads to keep it from blocking, but
>> somehow the active connection should become inactive while the
>> processing thread does it's work, so that another connection can be
>> processed in the meantime. Is that how it works? So twisted's event loop
>> (reactor) does the same work as the OS's scheduler?
>>
>
> In Twisted, code all runs in the main thread, so the way handling multiple
> connections works is that control is frequently returned to the reactor.
>
> So if a new connection is received, a protocol instance is made and its
> "connectionMade" method called, and as soon as that is done the reactor goes on
> to the next thing. That might be another connectionMade for a new connection,
> or it might be delivering a few bytes to a protocol (via calling dataReceived on
> it). So there a lots of events, but the event handlers (i.e. things like
> methods on Protocol subclasses) don't block, so they return control very
> quickly.
>
> In threaded network code typically you frequently do "sock.recv()" and similar,
> to block. In Twisted you just omit that; your protocol's dataReceived will be
> called whenever there's some data. This frees up the main thread to be doing
> other work when there's nothing happening on an individual connection. So
> that's how multiple connections can be handled concurrently in a single thread
> in Twisted.
>
> The only rule is that you cannot block in the main thread. If you want to do
> something blocking, find a different way to do it (or if you have to, do it in a
> thread and let the main thread continue, but at least for network I/O that's
> never necessary in Twisted).
>
> Does that help?
>
> -Andrew.
>
>
Yes Andrew, thanks! So keeps a list of all connections around and does
something with them when there is something to be done. So if I have a
shared list, only one connection actually access it at a time, thus
mutexes don't need to be used, but if something needs to be atomic, like
one message depending on the answer of another, I would use deferred
objects, If I need to to some heavy I/O (other than network) or lengthy
processing I would use a thread that returns a deferred, correct? So
like in code that calls an external program and I need to do something
when it is done, I would use twisted.internet.utils.getProcessValue(),
get a deferred and add a callback to do whatever I need to do when it is
done. Ok, I get it better now :-)
Thanks for all your help (Andrew and others)!!
Gabriel
More information about the Twisted-Python
mailing list