[Twisted-Python] Learning about IPushProducer

[Rutt, Benjamin]

    Sorry, your question was big and challenging to approach.

Understood, thanks.  I should have pared it down.

    This loop:
    
    >	        while not self.paused and self.howmany >
    >self.sent_already:
    >	            chunksz = min(maxchunksz, self.howmany -
    >self.sent_already)
    >	            self.consumer.write('x' * chunksz)
    >	            self.sent_already += chunksz
    
    is a bit atypical, I think.  The reason it is eventually stopping
    is that your code is being invoked re-entrantly by the consumer as
    soon as it decides its buffer is full.  I'm not sure the loop is
    /wrong/, but it is a bit surprising.  You don't need a mutex here,
    since it's single threaded, but you do need to be aware that your
    code can be re-entered within a single thread.
    
    Does that answer your questions?

Yes, thank you, it answers some of them but it raises more :).  I see,
it's self.consumer.write(...) that ends up calling the pause method.  So
that's how it's reentrant.  I knew twisted wasn't multithreaded in this
case.  Makes sense.

I have since made my resumeProducing() code loop for up to 100 (or 500,
or 1000 etc.) iterations and then return.  Thus I'm treating
resumeProducing as if it should "produce a chunk larger than 1 byte but
smaller that the whole dataset" then return.  How is this approach, does
it better match the intention of the producer/consumer system vs. the
"loop forever until paused" scheme I had earlier?

A final question -- is it safe for my code to ever call pauseProducing,
stopProducing, resumeProducing directly?  Or that will mess up the
balance of the twisted universe?  (I'm thinking that the calls to these
3 methods should only originate "from within twisted" in case the whole
producer/consumer system is tracking how many times it calls each one
etc.)  I don't plan on calling them directly, but want to critically
review in an informed way my colleague's code which does this.

Thank you so much for the response!

Benjamin Rutt