[Twisted-Python] sending large files from web2 http server

[David Reid]

On Thu, Mar 5, 2009 at 7:45 AM, Jean-Paul Calderone <exarkun at divmod.com>wrote:

> On Thu, 05 Mar 2009 16:20:27 +0100, Markus Wanner <markus at bluegap.ch>
> wrote:
>
>> Hi,
>>
>> I'm trying to stream longish data via web2, but experience sudden stalls
>> in data transfer, followed by a connection abort after a certain
>> timeout. I can't completely reproduce the issue, yet, but figured that
>> the size of the blocks I'm returning via the stream's read method
>> affects the failure rate (an stream.IByteStream implementation).
>>
>> I've tried with chunk sizes between 1 MiB down to 256 bytes. At that
>> rate, I suddenly get the following unhandled errors, which don't seem to
>> have to do much with my code.
>>
>
> I haven't looked at how web2 handles IByteStream providers, but my first
> guess would be that this is an example of a somewhat common bug where
> Deferreds are chained to an arbitrary length based on application data
> and when there's too much application data, the chain gets too long (a
> limit imposed by how much recursion is possible when unwinding the chain),
> and you get this failure.


I think this probably has something to do with the fact the IByteStream.read
may optionally return a Deferred making it quite easy to write code that
accidentally chain Deferreds to infinity and beyond.  Such as if you're
calling IByteStream.read in the callback of another Deferred and returning
the result.  Sometimes the read method will return a Deferred, sometimes
it'll just return your data.  Without reading the implementation of the
provider you're using it's not easy to know which it'll do when.

While the specific bug you're encountering may or may not be in your
application code (I haven't read it, please post a minimal example if you
can) it illuminates a fundamental flaw of the twisted.web2.stream APIs.
(It's a flaw we understand, and there is a ticket I can't find right now
that documents how it should work there may even be a branch dialtone worked
on that could possibly fix it.)

If this email doesn't help you find your problem please do post a minimal
example and I promise I'll read it today.

-David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090306/8663a686/attachment.html>