[Twisted-web] making streams optional in web2

[glyph at divmod.com]

A few days ago, JP showed me some benchmarks of web2 vs. CherryPy hosting a trivial WSGI application.

It turns out it's 4x slower, clocking in only ~250 req/sec on a middle-spec Opteron, as opposed to CherryPy's ~1000.  From previous benchmarks, I happen to know that Apache can do ~3000 req/sec on that same hardware, making web2 well over 10x slower.

I didn't look at the profile too closely, but preliminarily it seemed clear that the major bottleneck at this point was the creation and invocation of tons of Deferreds in _NotifyingProducerStream, while generating the response.

Considering that streams aren't used at all in the actual HTTP protocol implementation (and thank goodness for that, given this discovery) it strikes me that the use of them in the response-generating API ought to be more optional, for cases where efficiency is important.  Like, for example, DAV, which also seems to be making embarrassingly heavy use of the streams API - I haven't benchmarked that yet, but I'm sure it will be an adventure if I do :).

I'm not sure I totally understand the use of streams here, anyway.  I notice that Request purports to implement IConsumer, but I don't see any of IConsumer's methods actually present on Request.  I'd prefer it if it actually did implement IConsumer and I could just write to that as an API.

Finally, a minor nit; can we rename the various "http.py"s?  I'd rather a naming convention than subdirectories anyway, http_channel.py rather than channel/http.  Right now we have web2/http.py, web2/channel/http.py, and web2/dav/http.py.  I can only see this problem getting worse.