[Twisted-Python] Sending large files over network with perspective broker

[Gabriele Lanaro]

After putting some strategical sleeps seems that the problem is that the
application is the server and the client at once, I think that this
generates a "loop" in the mainloop.

2010/5/23 Gabriele Lanaro <gabriele.lanaro at gmail.com>

> Thank you very much for your responses, the problem seems to appear when in
> my tests I send the file "to myself", when I send files over the network,
> the things goes well.  I suspect (it's just a suspect) that the code spawns
> too much deferreds too much fast, causing the loop not to complete (or
> something like that).
>
> Anyway I think I'll follow your suggestion and I'll end splitting up the
> upload service and the control/authorization one, since the code can grow up
> too complex and performance is a requirement.
>
> - Gabriele
>
> 2010/5/23 David Bolen <db3l.net at gmail.com>
>
> Gabriele Lanaro <gabriele.lanaro at gmail.com> writes:
>>
>> > The problem of this approach is that this blocks my GUI, I can't figure
>> out
>> > why because I'm just generating deffereds so it souldn't block.
>>
>> Just using deferreds won't help unless you still manage to return
>> control back up the chain to the main event loop.  I suspect something
>> must be blocking somewhere, though hard to say from the pseudo-code.
>> Most likely a few judiciously placed logging statements would let you
>> see where, or at least verify that you are not returning to the main
>> event loop during the transfer.
>>
>> I will say that chunking up a large transfer through individual PB
>> requests adds a bit of overhead for a large stream, and unless you
>> implement some sort of windowing protocol, can hurt performance due to
>> the latency needed to wait for the ACK from the server for each chunk.
>>
>> I had what appears to be a similar requirement in terms of
>> transmitting a large file (A/V files to be published) as part of an
>> overall PB session, and decided to separate it out to its own file
>> upload server component coordinated through the PB session.
>>
>> http://twistedmatrix.com/pipermail/twisted-python/2007-July/015738.html
>> has some further details on what I ended up doing.  Perhaps an approach
>> along these lines would work for you as well.
>>
>> -- David
>>
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100523/116f3262/attachment.html>