[Twisted-Python] Sending large files over network with perspective broker

[Gabriele Lanaro]

Finally I managed to solve myself this obscure bug. I put the
reactor.iterate() call before spawning new callbacks, in this way I force
the mainloop to complete the cycle. The pseudocode would be modified in this
way:

Client()
    proceed_sending():

          chunk_tot = CHUNK_TOT
          chunk_no = 0

          def send(_)
                if chunk_no == CHUNK_TOT: return
                else:
                     ... read data...
                    reactor.iterate()
<---------------------------------------------------------- THIS LINE
                    d = filesender.callRemote("send_
chunk", secret, chunk_no, data)
                    d.addCallback(send)



2010/5/23 Gabriele Lanaro <gabriele.lanaro at gmail.com>

> After putting some strategical sleeps seems that the problem is that the
> application is the server and the client at once, I think that this
> generates a "loop" in the mainloop.
>
> 2010/5/23 Gabriele Lanaro <gabriele.lanaro at gmail.com>
>
> Thank you very much for your responses, the problem seems to appear when in
>> my tests I send the file "to myself", when I send files over the network,
>> the things goes well.  I suspect (it's just a suspect) that the code spawns
>> too much deferreds too much fast, causing the loop not to complete (or
>> something like that).
>>
>> Anyway I think I'll follow your suggestion and I'll end splitting up the
>> upload service and the control/authorization one, since the code can grow up
>> too complex and performance is a requirement.
>>
>> - Gabriele
>>
>> 2010/5/23 David Bolen <db3l.net at gmail.com>
>>
>> Gabriele Lanaro <gabriele.lanaro at gmail.com> writes:
>>>
>>> > The problem of this approach is that this blocks my GUI, I can't figure
>>> out
>>> > why because I'm just generating deffereds so it souldn't block.
>>>
>>> Just using deferreds won't help unless you still manage to return
>>> control back up the chain to the main event loop.  I suspect something
>>> must be blocking somewhere, though hard to say from the pseudo-code.
>>> Most likely a few judiciously placed logging statements would let you
>>> see where, or at least verify that you are not returning to the main
>>> event loop during the transfer.
>>>
>>> I will say that chunking up a large transfer through individual PB
>>> requests adds a bit of overhead for a large stream, and unless you
>>> implement some sort of windowing protocol, can hurt performance due to
>>> the latency needed to wait for the ACK from the server for each chunk.
>>>
>>> I had what appears to be a similar requirement in terms of
>>> transmitting a large file (A/V files to be published) as part of an
>>> overall PB session, and decided to separate it out to its own file
>>> upload server component coordinated through the PB session.
>>>
>>> http://twistedmatrix.com/pipermail/twisted-python/2007-July/015738.html
>>> has some further details on what I ended up doing.  Perhaps an approach
>>> along these lines would work for you as well.
>>>
>>> -- David
>>>
>>>
>>> _______________________________________________
>>> Twisted-Python mailing list
>>> Twisted-Python at twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100523/28d7220b/attachment.html>