[Twisted-Python] Re: more thoughts on resumable async data flows

[Clark C. Evans]

On Thu, Apr 03, 2003 at 03:02:04PM +0200, Philippe Lafoucrière wrote:
| how do you use it with adbapi ?? I mean, how do you use callbacks
| with your flow ?
|
| How does it work if a bloc has several inputs ?
| I want to do this :
| 
| DS1 ---> FS1 ---+
|                 +--->  FS3
| DS2 ---> FS2 ------->  FS3

If your queries are returning small results, ie, 
non-incremental, I would just set this up as a 
chain of Deferreds and not bother with flow as
it might be overkill for your needs.

In other words, in the twisted.enterprise.adbapi
when you run a query, it returns a deferred.  So
DS1 and DS2 would be those query results.

Your FS1 and FS2 should be deferred operations
as well, but fired via the 'addCallback' on 
DS1 and DS2 respectively.

Then FS3 is a DeferredList taking FS1 and FS2.

...

If you can use generators, for more complicated flows, 
etrepum's (Bob Ippolito's) approach may be better approach
to this, although I'm not sure how it would handle more
'fluid' flow through the process.  I need to understand
what he's doing better.   At first I thought his approach
is very different than mine, but I think (although I'm
not quite yet able to explain) that they are isomorphic;
only that I'm using a temp stack and he's using the 
event queue via a linked-list. 

...

As for the stuff in flow.py, right now it uses a thread
for each database connection.  Eventually for PostgreSQL,
I'd like to move over to a non-threaded driver.  Gerhard
was working on such a thing as I remember.  Also, it may
be renamed since extrepum was using "flow.py" before me.

Anyway, the database connection is done through a 
QueryIterator which basically sends a set of rows
(via fetchmany) on to the next stage.  So, the connection
from DS1 to FS1 and DS2 to FS2 could be done using 
a simple flow for each.  The problem in the flow module 
has to do with merging the two flows.  This can be done 
by using the 'context'.

In short, it would work, but I'm not sure that it's
your best option; and I'm still not quite happy with
flow.py myself (although it is in production).

Clark