[Twisted-Python] Re: Send many large files with PB

[Justin Mazzola Paluska]

On Wed, May 17, 2006 at 08:41:21AM +0200, Nicola Larosa wrote:
> In my experience, sending big files over PB takes way too much time. This
> is due to the serialization-deserialization process involved. Paging avoids
> blocking, which is good, but it still takes much more than sending the
> files as-is.

How bad is the slow down?  Or, to ask the question another way, how
much CPU will the process actually take?  

I ask because the machines in question are being used as file servers
for streaming applications like video and whatnot, so having Twisted
suck up all of the CPU may disrupt the other streams.  (We're not
using Twisted to actually serve the files, but rather as the
application framework for various web- and GUI- based management
clients.  The copy data feature would be the first time we're moving
lots of data with Twisted.)

> > - Finally, should I be doing something completely different?
> >   Normally, outside of my application, I'd just use rsync, scp, or
> >   some such.  However, the users of this application don't know how to
> >   use these tools.  I can't spawn these programs without getting into
> >   authentication issues between the machines.  Doing this within
> >   Twisted seems like a good idea because the machines are already
> >   authenticated to each other through PB, but I could be wrong.
> 
> You could send the files over an HTTP connection, avoiding the
> serialization overhead. Setting up HTTP clients and servers is very easy in
> Twisted, as you surely know.

Indeed! Using HTTP is appealing because it is closer to just stuffing
bits in a socket from an efficiency standpoint.  Is there a way of
passing a RemoteReference to an HTTP server?  Or is the best thing to
do just use the PB to send a URL to the DEST server?
   --Justin