[Twisted-Python] Sending large files over network with perspective broker

[David Bolen]

Gabriele Lanaro <gabriele.lanaro at gmail.com> writes:

> The problem of this approach is that this blocks my GUI, I can't figure out
> why because I'm just generating deffereds so it souldn't block.

Just using deferreds won't help unless you still manage to return
control back up the chain to the main event loop.  I suspect something
must be blocking somewhere, though hard to say from the pseudo-code.
Most likely a few judiciously placed logging statements would let you
see where, or at least verify that you are not returning to the main
event loop during the transfer.

I will say that chunking up a large transfer through individual PB
requests adds a bit of overhead for a large stream, and unless you
implement some sort of windowing protocol, can hurt performance due to
the latency needed to wait for the ACK from the server for each chunk.

I had what appears to be a similar requirement in terms of
transmitting a large file (A/V files to be published) as part of an
overall PB session, and decided to separate it out to its own file
upload server component coordinated through the PB session.

http://twistedmatrix.com/pipermail/twisted-python/2007-July/015738.html
has some further details on what I ended up doing.  Perhaps an approach
along these lines would work for you as well.

-- David