[Twisted-Python] looping in a non-blocking way

[Eugene Coetzee]

Hi everybody.

I have an application where I have  several polling protocol instances 
servicing an array of COM ports driven by  timers using 
reactor.callLater with typically 50 millisecond timing constants.  I'm 
also using  FTP client protocol instances to concurrently  refresh 
locally cached data from ftp servers. These files may be quite large and 
during the text parsing routines I end up with blocking loops which are 
undesirable.

Text files my be parsed in a loop with something like:

def importFromText(self,data):
               datalines=data.split("\r\n")
                for line in datalines:
                        arrin=[]
                        arrin.insert(0,line[0:10])
                        etc....

For large files the above blocks - starving the other COM port protocols 
drivers of processing time.

I have tried to "yield" to the reactor by calling reactor.iterate() in 
the above loop trying to do what you would normally accomplish with 
sleep(0) in a multithreaded  application. This solution seemed to 
partially solve the problem but I'm experiencing intermittent  stability 
problems with my polling protocol drivers that are normally quite 
robust  simply hanging up.

I'm now considering rewriting the above loop as a recursive function 
using reactor.callLater(0,recursiveTexParsingFunction(params)).

I'm aware of twisted.flow but the examples given are a bit sparse and 
obscure and having tried something like  
http://twistedmatrix.com/documents/current/api/twisted.flow.flow.html - 
it appears I don't understand how to apply twisted.flow  
(iterators/coroutines) solve my particular problem.

I hope I'm missing something obvious and more elegant since the above 
possible solutions seems like a lot of rework to solve a very simple 
problem.

Any thoughts/advise would be much appreciated.

regards,

Eugene Coetzee                  


Web                 -> www.reedflute.com
===============================================