[Twisted-Python] sending large files from web2 http server

[Jean-Paul Calderone]

On Thu, 05 Mar 2009 16:20:27 +0100, Markus Wanner <markus at bluegap.ch> wrote:
>Hi,
>
>I'm trying to stream longish data via web2, but experience sudden stalls
>in data transfer, followed by a connection abort after a certain
>timeout. I can't completely reproduce the issue, yet, but figured that
>the size of the blocks I'm returning via the stream's read method
>affects the failure rate (an stream.IByteStream implementation).
>
>I've tried with chunk sizes between 1 MiB down to 256 bytes. At that
>rate, I suddenly get the following unhandled errors, which don't seem to
>have to do much with my code.

I haven't looked at how web2 handles IByteStream providers, but my first
guess would be that this is an example of a somewhat common bug where
Deferreds are chained to an arbitrary length based on application data
and when there's too much application data, the chain gets too long (a
limit imposed by how much recursion is possible when unwinding the chain),
and you get this failure.

Alternatively, the bug I describe above might somehow have gotten into
your application code, rather than into Web2.  As you say, the stack
trace doesn't point to your code, but it doesn't point to Web2 code
either, and it's common in cases like this for the traceback to just
point at the implementation of Deferred, since that's where the bug
(or lack of feature, whatever) *really* is.

The main focus of web development in Twisted now is Twisted Web, not
Twisted Web2.  However, if you can supply a patch which fixes this
(along with unit tests, please :), then we'd be happy to apply it.

James Knight or David Reid might have a suggestion about where in the
code to start looking for the problem.

Jean-Paul