[Twisted-Python] Question on push/pull producers inter-workings, was : "Is there a simple Producer/Consumer example or tutorial?"

[Jean-Paul Calderone]

On Fri, 18 Apr 2008 09:57:35 +0200, Gabriel Rossetti <mailing_lists at evotex.ch> wrote:
>Hello everyone,
>
>I am re-posting these questions with a different title, since they have 
>ventured away from the original question
>
>I have a question though, I was looking at the way all this works by using a 
>debugger, and I noticed that in twisted.internet.abstract.py in 
>registerProducer() there is the following :
>
>if not streaming:
>   producer.resumeProducing()
>
>Why is this done only for the pull producer? Shouldn't it also be called for 
>the push producer since to have the data sent one has to call either 
>self.transport.write() or resumeProducing() anyways? If you look at :
>
>http://itamarst.org/writings/OSCON03/twisted_internet-112.html
>
>it does :
>
>transport.registerProducer(self, 1)
>       self.produce()

So, this is a streaming producer.  It starts producing right away without
having resumeProducing called on it.

>
>thus starting the writing process and in the pull producer :
>
>http://itamarst.org/writings/OSCON03/twisted_internet-111.html
>
>it doesn't need to start the writing process explicitly since it's started 
>when the producer is registered.

This is a non-streaming producer.  It doesn't do anything until something
calls resumeProducing on it.

>
>Oh, and also, since from what I see in the code the only difference with a 
>push and pull producer is that the push producer is paused if the data being 
>written/sent is very large (to let the reactor breath and process other 
>things), if it isn't larger than the buffer it behaves like a pull producer, 
>correct? If so, then why have both? Am I mixed up again?

As you say, they behave differently when there is a large amount of data.
However, this is more about the source of the data than where it ends up.
For example, if you have a large string and you want to producer it to a
transport, you probably want a pull producer, because there's no events
which will signal that you can send some more of the string *except* for
the reactor deciding that it is ready for some more.  So that's how you
should decide which of these you want to write - if the consumer is the
only event source involved, as in the large string case, then you want a
pull producer (streaming = False); if the producer itself is event-driven
in its ability to provide data, then you want a push producer.

Jean-Paul