[Twisted-Python] newbie: groking flows

[Clark C. Evans]

On Wed, Oct 08, 2003 at 07:21:02PM -0700, Nesbitt, Steve wrote:
| During the regeneration process, which
| can take multiple seconds, all lookup and additional regeneration requests
| should block until completion of the regeneration. Once completed, blocked
| requests should resume.
| 
| Based on my reading I think this is what should do:
| 1) Periodically during a lookup call a flow.Cooperate event to allow other
| lookup events from other requests to be handled.

Right.

| 2) To avoid a regeneration event in the midst of a lookup, I should make
| sure that a flow.Cooperate event is *not* issued at any time during a
| critical part of the lookup process ( In other words one can consider the
| code executed between two flow.Cooperate statements as a transaction)

Right.

| 3) When I start a regeneration event I should wrap the regeneration method
| in a flow.Block controller.

flow.Block is primarly there for testing; I should remove it from
the public interface.  Basically, it does just that, it blocks
*everything* -- which is, of course, not what you want.

You should instead use flow.Deferred( <generator> ) to create
your data flows, when you run flow.Deferred it:

  1. schedules your generator to be run using reactor.callLater(0, ...)
     so, flow.Deferred returns almost immediately and does not run
     your code till the current code finishes running;

  2. for each time you return flow.Cooperate, it reschedules your
     generator to be executed later, ie, reactor.callLater(0, ...)

  3. after your generator finishes, the deferred that was created
     by flow.Deferred runs its callbacks; you should probably install
     an error handler here, ie  flow.Deferred(<generator)>.addErrback(...)

I hope this helps.

...

| 1) Is my basic understanding correct? In particular am I guaranteed correct
| behavior using the above outline?

You are close; s/flow.Block/flow.Deferred/

| 2) Am I correct in believing that wrapping a method in flow.Block actually
| executes the method? That is, the statement flow.Block( foo( bar ) )
|    executes foo.bar immediately?

Not immediately, but close enough (it is put on the reactor's queue
to be executed shortly)

| 3) How do I access the results of flow.Block( foo( bar ) )? In particular
| how do I see the return result or any exceptions tr\hrown by foo?

def good(results):
    print "good", results
def bad(failure):
    print "bad", failure

d = flow.Deferred()
d.addCallback(good)
d.addErrback(bad)


...

If you _really_ want to use flow.Block, then flow.Block executes
the function immediately and returns a list containing the results.

| 4) Am I correct in believing that while flow.Block( foo(bar) ) is executing
| any incoming requests will be prevented from executing?

For flow.Block, yes.  For flow.Deferred, no.

| 5) Is it possible to defer a blocking method? This covers the scenario where
| two regeneration events occur within the same time frame.

Yes, use flow.Deferred  

To handle your case where there is a "regeneration" happening,
suppose you have a object 'gList' which gets regenerated periodically.
Your generator could do...

   def gen():
       while gList.generating():
           yield flow.Cooperate()
       # access the gList that is not generating

Best,

Clark