[Twisted-Python] shared list in factory, mutexes?

[Gabriel Rossetti]

Andrew Bennetts wrote:
> Gabriel Rossetti wrote:
> [...]
>   
>> think what I don't get is that with thread network code, each thread is  
>> an individual self-contained unit that does whatever processing it needs  
>> to do however long it may need to take, and the OS takes care of  
>> scheduling each thread, but with twisted, the main thread executes each  
>> connection one by one sequentially and thus if one of them needs to to  
>> some lengthy processing, it will block all other connections from being  
>> processed, so you have to use threads to keep it from blocking, but  
>> somehow the active connection should become inactive while the  
>> processing thread does it's work, so that another connection can be  
>> processed in the meantime. Is that how it works? So twisted's event loop  
>> (reactor) does the same work as the OS's scheduler?
>>     
>
> In Twisted, code all runs in the main thread, so the way handling multiple
> connections works is that control is frequently returned to the reactor.
>
> So if a new connection is received, a protocol instance is made and its
> "connectionMade" method called, and as soon as that is done the reactor goes on
> to the next thing.  That might be another connectionMade for a new connection,
> or it might be delivering a few bytes to a protocol (via calling dataReceived on
> it).  So there a lots of events, but the event handlers (i.e. things like
> methods on Protocol subclasses) don't block, so they return control very
> quickly.
>
> In threaded network code typically you frequently do "sock.recv()" and similar,
> to block.  In Twisted you just omit that; your protocol's dataReceived will be
> called whenever there's some data.  This frees up the main thread to be doing
> other work when there's nothing happening on an individual connection.  So
> that's how multiple connections can be handled concurrently in a single thread
> in Twisted.
>
> The only rule is that you cannot block in the main thread.  If you want to do
> something blocking, find a different way to do it (or if you have to, do it in a
> thread and let the main thread continue, but at least for network I/O that's
> never necessary in Twisted).
>
> Does that help?
>
> -Andrew.
>
>   
Yes Andrew, thanks! So keeps a list of all connections around and does 
something with them when there is something to be done. So if I have a 
shared list, only one connection actually access it at a time, thus 
mutexes don't need to be used, but if something needs to be atomic, like 
one message depending on the answer of another, I would use deferred 
objects, If I need to to some heavy I/O (other than network) or lengthy 
processing I would use a thread that returns a deferred, correct? So 
like in code that calls an external program and I need to do something 
when it is done, I would use twisted.internet.utils.getProcessValue(), 
get a deferred and add a callback to do whatever I need to do when it is 
done. Ok, I get it better now :-)

Thanks for all your help (Andrew and others)!!
Gabriel