[Twisted-Python] Implementing streaming (broadcast) TCP servers

[Adam Atlas]

On 28 Jun 2007, at 09.32, Carl Zmola wrote:
> 3) CPU -- Profile your application and see where the time is being
> spent.  You need to be careful about memory copying and allocation.
> Python normally handles this well, so this probably isn't the problem.

Ah, good point. I'll try that.

> OOPS,  I just realized that you said the same data to multiple  
> clients.
> Is the data the same at the same time, meaning the clients are all
> getting the same data simultaneously?  If so, if you have any control
> over the network, then UDP Multicast can be a big win.  You would only
> be sending out one stream of data and everyone would hear it.

It is indeed being sent at the same time, but unfortunately I'm not  
designing the protocol. I'm implementing an existing one that only  
runs over HTTP.

> Also, I would look at making sure that you are not copying data  
> around much.

I was definitely paying attention to that. I'm using Python's  
`buffer` type when possible to prevent any duplication of data;  
ideally, there's only one real copy of it in memory, and a bunch of  
buffers pointing to it... then it can be garbage collected once it's  
been sent to all clients.

> I must admit that my Knowledge of all the twisted classes that are
> available is limited, so I don't know what class to use but I would
> suggest dropping in to the low level classes below web2 since you will
> need a little more control for this than normal web applications.

I had similar thoughts... I was thinking, once I had the connection  
established, I could try to bypass web2's IByteStream stuff and try  
to use transport.write directly. Do you think that would help? It's  
less runloopy overhead, at least.