[Twisted-Python] Learning about IPushProducer

[Jean-Paul Calderone]

On Mon, 12 Mar 2007 17:46:54 -0400, "Rutt, Benjamin" <benjamin.rutt at gs.com> wrote:
>    Sorry, your question was big and challenging to approach.
>
>Understood, thanks.  I should have pared it down.
>
>    This loop:
>
>    >	        while not self.paused and self.howmany >
>    >self.sent_already:
>    >	            chunksz = min(maxchunksz, self.howmany -
>    >self.sent_already)
>    >	            self.consumer.write('x' * chunksz)
>    >	            self.sent_already += chunksz
>
>    is a bit atypical, I think.  The reason it is eventually stopping
>    is that your code is being invoked re-entrantly by the consumer as
>    soon as it decides its buffer is full.  I'm not sure the loop is
>    /wrong/, but it is a bit surprising.  You don't need a mutex here,
>    since it's single threaded, but you do need to be aware that your
>    code can be re-entered within a single thread.
>
>    Does that answer your questions?
>
>Yes, thank you, it answers some of them but it raises more :).  I see,
>it's self.consumer.write(...) that ends up calling the pause method.  So
>that's how it's reentrant.  I knew twisted wasn't multithreaded in this
>case.  Makes sense.
>
>I have since made my resumeProducing() code loop for up to 100 (or 500,
>or 1000 etc.) iterations and then return.  Thus I'm treating
>resumeProducing as if it should "produce a chunk larger than 1 byte but
>smaller that the whole dataset" then return.  How is this approach, does
>it better match the intention of the producer/consumer system vs. the
>"loop forever until paused" scheme I had earlier?

That's more like what I expect to see in a resumeProducing, yea.  Of course,
paused might still get set before your loop decides to exit, if you are no
longer checking paused.  This isn't disasterous, but it means some
information about what a good amount of stuff to keep in memory is.

I think the most common thing for a resumeProducing implementation to do is
build up a chunk of data and then call write on the consumer just once.  So
it doesn't end up mattering if you get paused or not, since you're done any
way by the time that happens.

I think you have the basic idea of how producers should work, though, and
now I'm just blabbing. :)

>
>A final question -- is it safe for my code to ever call pauseProducing,
>stopProducing, resumeProducing directly?  Or that will mess up the
>balance of the twisted universe?  (I'm thinking that the calls to these
>3 methods should only originate "from within twisted" in case the whole
>producer/consumer system is tracking how many times it calls each one
>etc.)  I don't plan on calling them directly, but want to critically
>review in an informed way my colleague's code which does this.

The consumer you register the producer with basically gets control of those
methods for as long as the producer remains registered.  They're there for
it to communicate its needs with the producer, so having any other code call
them is probably bad in general, although it might work in specific cases.

>
>Thank you so much for the response!
>

No problem. :)

Jean-Paul