[Twisted-Python] HTTP client should be more tolerant?

[Markus Stenberg]

James Y Knight <foom at fuhm.net> writes:
> On Feb 3, 2005, at 10:36 AM, Markus Stenberg wrote:
<snip>
>> Some UNIXish servers, while breaking RFC, seem to work on
>> browsers. I'd say
>> following this advice would be good, and therefore tuned my twisted
>> http
>> client to be 'tolerant' (all major and minor browsers that I am
>> aware of
>> are).
> OMG. *What* evil server does that?

Offhand I can remember only some old ones, i.e. they have been fixed since
(some 'minimal web servers' in assorted languages were guilty of this,
mostly in the HTTP/1.0 and before days). But many, many CGI scripts are
broken that way still (because if you code on UNIX, and dump headers using
say, perl's print, you don't naturally use \r\n unless you really know the
standard, and unfortunately many CGI authors don't seem to. And web servers
usually dump the results out directly => corruption spreads.

>> I'm also considering implementing HTTP/1.1 client. Has anyone done
>> anything about it? Thoughts? 'Give up, bad idea'?
> Thought about it, but still working on the new HTTP server. (see
> twisted.web2 in svn trunk). Hopefully a new client could reuse much of
> the work for the new server (esp. header parsing/generation).

One could imagine that given correct header parsing/generation and request
parsing code (chunked de/encoding for example needs to be implemented in
both ends), it would be trivial. I'll take a look ;)

>> I also rewrote the proxy for my local use a bit, mostly to make it more
>> component-oriented AND correct (currently HTTP/1.1 clients do not get
>> persistent service, which is inefficient). Unfortunately changes are
>> mixed
>> among some other code, but has there been any work on proxy recently?
>> I might have interest in submitting patch or two to make it more
>> correct in
>> some things.
> A proxy should only be a small glue layer between the server and
> client, so once those two are working well, a correctly operating
> (non-caching) proxy should be pretty easy. A caching proxy, of course,
> is a whole other project.

True. The tricky part is making it modular so that it can be extended
without say, subclassing whole set of classes (as is needed to change the
current twisted.web.proxy). Obviously, less clean hack of mutating the
static variables of classes that point to subclasses works to limited
degree too, but I avoid stuff like that.

> James

-Markus