[Twisted-Python] Sending large files over network with perspective broker

[Gabriele Lanaro]

Thank you very much for your responses, the problem seems to appear when in
my tests I send the file "to myself", when I send files over the network,
the things goes well.  I suspect (it's just a suspect) that the code spawns
too much deferreds too much fast, causing the loop not to complete (or
something like that).

Anyway I think I'll follow your suggestion and I'll end splitting up the
upload service and the control/authorization one, since the code can grow up
too complex and performance is a requirement.

- Gabriele

2010/5/23 David Bolen <db3l.net at gmail.com>

> Gabriele Lanaro <gabriele.lanaro at gmail.com> writes:
>
> > The problem of this approach is that this blocks my GUI, I can't figure
> out
> > why because I'm just generating deffereds so it souldn't block.
>
> Just using deferreds won't help unless you still manage to return
> control back up the chain to the main event loop.  I suspect something
> must be blocking somewhere, though hard to say from the pseudo-code.
> Most likely a few judiciously placed logging statements would let you
> see where, or at least verify that you are not returning to the main
> event loop during the transfer.
>
> I will say that chunking up a large transfer through individual PB
> requests adds a bit of overhead for a large stream, and unless you
> implement some sort of windowing protocol, can hurt performance due to
> the latency needed to wait for the ACK from the server for each chunk.
>
> I had what appears to be a similar requirement in terms of
> transmitting a large file (A/V files to be published) as part of an
> overall PB session, and decided to separate it out to its own file
> upload server component coordinated through the PB session.
>
> http://twistedmatrix.com/pipermail/twisted-python/2007-July/015738.html
> has some further details on what I ended up doing.  Perhaps an approach
> along these lines would work for you as well.
>
> -- David
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100523/0b780667/attachment.html>