[Twisted-Python] Quan Nguyen: Twisted Python questions

[Glyph]

On Jun 12, 2012, at 1:53 PM, Quan Nguyen wrote:

> Hello,
> 
> I am new to Twisted. 
> I am extending the twisted Echo server example by sending the data that the server receives from telnet into a child process that processes the data. The function outReceived in censoringProtocol is called when the child writes the result and the result is received. The process of passing data from parent to child and back to parent is done by having a while loop of stdin.readline() and stdout.write() in the child process. 
> Specifically, I generate a deferred object each time I write to the child using the function sendData (which then calls writeToChild) and use a queue to manage the deferred objects. I add a callBack (printData) that will broadcast the result to all instances of Echo when outReceived is called. 
> 
> I have two questions: 
> 1/ Is this the right approach to communicate with the child process in order to achieve what I want? Are there any gotcha's?

This sounds pretty much fine, except for the caveat you already asked about:

> 2/ This server will be exposed to high loads. There will be many instances of Echo sending data to the child at a high rate. Are there any limitations that I have to watch for, such as the limit on child process's pipe sizes or anything else?

You might want to investigate the producer/consumer API, as recently blogged about by our marvelously productive summer intern, ashfall: <http://ashfall.github.com/blog/2012/05/29/twisted-producer-and-consumer-system/>.  As currently written, even if your subprocess backs up, you will buffer data to it without any limits.

(The consumer you want, by the way, is censoringProtocol().transport.pipes[0], in case that isn't entirely clear; processes need to be documented a little better.)

You may have to write a little bit of tricky code to multiplex the consumer, if you want to have multiple "echo" client connections using the same subprocess.  If it were a 1<->1 relationship you could just register the echo protocol's transport as the producer for the stdin pipe on the processing protocol, but spawning that many processes is probably wasteful.

-glyph