[Twisted-Python] Re: Send many large files with PB

[Nicola Larosa]

Nicola Larosa:
>> In my experience, sending big files over PB takes way too much time. This
>> is due to the serialization-deserialization process involved. Paging avoids
>> blocking, which is good, but it still takes much more than sending the
>> files as-is.

Justin Mazzola Paluska:
> How bad is the slow down?  Or, to ask the question another way, how
> much CPU will the process actually take?  

100% CPU for all the time it takes. Serialization is CPU- and memory-intensive.


>> You could send the files over an HTTP connection, avoiding the
>> serialization overhead. Setting up HTTP clients and servers is very easy in
>> Twisted, as you surely know.

> Indeed! Using HTTP is appealing because it is closer to just stuffing
> bits in a socket from an efficiency standpoint.  Is there a way of
> passing a RemoteReference to an HTTP server?

I don't think so. There's no overlapping that I know of, between PB and HTTP.


> Or is the best thing to do just use the PB to send a URL to the DEST
> server?

That's what I was hinting at, yes. Of course you should separately take
care of any required authentication, authorization and encryption on the
HTTP connection.


-- 
Nicola Larosa - http://www.tekNico.net/

If you know you can love work, you're in the home stretch, and if you know
what work you love, you're practically there. -- Paul Graham, January 2006