[Twisted-Python] Implementing streaming (broadcast) TCP servers
Adam Atlas
adam at atlas.st
Wed Jun 27 14:33:16 EDT 2007
I'm working on a server that will be streaming the same data from a
source to many clients. It has to be invoked from an HTTP server, so
I'm using twisted.web2 (not twisted.web because it also needs to host
a WSGI application). In other words, calling GET on a certain path
results in the stream as the response, at which point I don't really
need any more HTTP functionality. I have it working... currently, the
basic process is: I have a class implementing IByteStream, an
instance of which is created for each client; it simply keeps a queue
of data to be sent, sending one piece with each call to read(), or a
sending a Deferred if the queue is empty. Meanwhile, the part of the
application that provides the data (a different server, receiving
data from a broadcaster) adds the data to all the clients' byte
stream queues, whenever it receives a piece of data.
Since efficiency is obviously important for a streaming server, I was
just wondering if anyone had any efficiency tips. I've tested it with
at most 64 concurrent clients. There doesn't seem to be any slowdown
from the clients' perspective (the data is coming in at the rate it
should), but server-side, CPU and memory usage start increasing
pretty quickly. Especially CPU -- with 64 clients, CPU usage peaked
at 26.8%, which is not acceptable. Any thoughts? Is there a better
way to handle the streaming process as I previously described?
- Adam
More information about the Twisted-Python
mailing list