[Twisted-Python] Question on push/pull producers inter-workings, was : "Is there a simple Producer/Consumer example or tutorial?"

[Gabriel Rossetti]

Jean-Paul Calderone wrote:
> On Fri, 18 Apr 2008 09:57:35 +0200, Gabriel Rossetti 
> <mailing_lists at evotex.ch> wrote:
>> Hello everyone,
>>
>> I am re-posting these questions with a different title, since they 
>> have ventured away from the original question
>>
>> I have a question though, I was looking at the way all this works by 
>> using a debugger, and I noticed that in twisted.internet.abstract.py 
>> in registerProducer() there is the following :
>>
>> if not streaming:
>>   producer.resumeProducing()
>>
>> Why is this done only for the pull producer? Shouldn't it also be 
>> called for the push producer since to have the data sent one has to 
>> call either self.transport.write() or resumeProducing() anyways? If 
>> you look at :
>>
>> http://itamarst.org/writings/OSCON03/twisted_internet-112.html
>>
>> it does :
>>
>> transport.registerProducer(self, 1)
>>       self.produce()
>
> So, this is a streaming producer.  It starts producing right away without
> having resumeProducing called on it.
>
>>
>> thus starting the writing process and in the pull producer :
>>
>> http://itamarst.org/writings/OSCON03/twisted_internet-111.html
>>
>> it doesn't need to start the writing process explicitly since it's 
>> started when the producer is registered.
>
> This is a non-streaming producer.  It doesn't do anything until something
> calls resumeProducing on it.
>
>>
>> Oh, and also, since from what I see in the code the only difference 
>> with a push and pull producer is that the push producer is paused if 
>> the data being written/sent is very large (to let the reactor breath 
>> and process other things), if it isn't larger than the buffer it 
>> behaves like a pull producer, correct? If so, then why have both? Am 
>> I mixed up again?
>
> As you say, they behave differently when there is a large amount of data.
> However, this is more about the source of the data than where it ends up.
> For example, if you have a large string and you want to producer it to a
> transport, you probably want a pull producer, because there's no events
> which will signal that you can send some more of the string *except* for
> the reactor deciding that it is ready for some more.  
What exactly do you mean by :

"there's no events which will signal that you can send some more of the 
string *except* for
the reactor deciding that it is ready for some more"?

When I looked at Twisted's code, the difference that I saw was that if a 
push producer is used, and if the data to be sent is bigger than a 
certain length, it calls producer.pauseProducing()

> So that's how you
> should decide which of these you want to write - if the consumer is the
> only event source involved, as in the large string case, then you want a
> pull producer (streaming = False); 
How can the consumer be an event source? The producer is the one sending 
the data, maybe I don't get what you mean by "event source".
> if the producer itself is event-driven
> in its ability to provide data, then you want a push producer.
>
> Jean-Paul
I thought the push producer worked like this : if the data is too big, 
send part of it and pause the rest, let the reactor breath some, and 
repeat. I thought the pull producer was basically like if no producer 
was used, one has to take care of any data splitting and send small 
parts when the consumer is ready. Is this not correct?

Gabriel