[Twisted-Python] Re: Re: more thoughts on resumable async data flows

[Philippe Lafoucrière]

> If your queries are returning small results, ie,
> non-incremental, I would just set this up as a
> chain of Deferreds and not bother with flow as
> it might be overkill for your needs.

No, this may be used with a very large amount of data.

 
> In other words, in the twisted.enterprise.adbapi
> when you run a query, it returns a deferred.  So
> DS1 and DS2 would be those query results.

I agree with that
 
> Your FS1 and FS2 should be deferred operations
> as well, but fired via the 'addCallback' on
> DS1 and DS2 respectively.

How do you fire it exactly ? Moreover, the tutorial about deferred isn't so
clear for me (twisted newbie).
 
> Then FS3 is a DeferredList taking FS1 and FS2.

Just a note : this system may be dynamic. I mean that the user can create
the "flow" he wants (with various datasources and other blocs).

> If you can use generators, for more complicated flows,
> etrepum's (Bob Ippolito's) approach may be better approach
> to this, although I'm not sure how it would handle more
> 'fluid' flow through the process.  I need to understand
> what he's doing better.   At first I thought his approach
> is very different than mine, but I think (although I'm
> not quite yet able to explain) that they are isomorphic;
> only that I'm using a temp stack and he's using the
> event queue via a linked-list.
> 

I don't really know generators. I'm going to take a look at like tomorow...


> As for the stuff in flow.py, right now it uses a thread
> for each database connection.  Eventually for PostgreSQL,
> I'd like to move over to a non-threaded driver.  Gerhard
> was working on such a thing as I remember.  Also, it may
> be renamed since extrepum was using "flow.py" before me.
> 
> Anyway, the database connection is done through a
> QueryIterator which basically sends a set of rows
> (via fetchmany) on to the next stage.  So, the connection
> from DS1 to FS1 and DS2 to FS2 could be done using
> a simple flow for each.  The problem in the flow module
> has to do with merging the two flows.  This can be done
> by using the 'context'.

3 flows  ? (1 for DS1 to FS1, 1 for DS2 to FS2, and one the merge the 2
others) 

I was thinking of using only one flow. It's weird to seperate this process
in several flow, isn't it ?

 
> In short, it would work, but I'm not sure that it's
> your best option; and I'm still not quite happy with
> flow.py myself (although it is in production).

thank you for your help
 
> Clark

Thank very much Clark !
Philippe