[Twisted-Python] HTTP client should be more tolerant?

[James Y Knight]

On Feb 3, 2005, at 10:36 AM, Markus Stenberg wrote:
> According to the HTTP RFCs, CRLF is the valid line termination
> sequence. However, the appendix (section 19.3 in old HTTP/1.1, too 
> lazy to
> dig up more recent one) states as follows:
>
>    The line terminator for message-header fields is the sequence CRLF.
>    However, we recommend that applications, when parsing such headers,
>    recognize a single LF as a line terminator and ignore the leading 
> CR.
>
> Some UNIXish servers, while breaking RFC, seem to work on browsers. 
> I'd say
> following this advice would be good, and therefore tuned my twisted 
> http
> client to be 'tolerant' (all major and minor browsers that I am aware 
> of
> are).

OMG. *What* evil server does that?

> I'm also considering implementing HTTP/1.1 client. Has anyone done 
> anything
> about it? Thoughts? 'Give up, bad idea'?

Thought about it, but still working on the new HTTP server. (see 
twisted.web2 in svn trunk). Hopefully a new client could reuse much of 
the work for the new server (esp. header parsing/generation).

> I also rewrote the proxy for my local use a bit, mostly to make it more
> component-oriented AND correct (currently HTTP/1.1 clients do not get
> persistent service, which is inefficient). Unfortunately changes are 
> mixed
> among some other code, but has there been any work on proxy recently?
> I might have interest in submitting patch or two to make it more 
> correct in
> some things.

A proxy should only be a small glue layer between the server and 
client, so once those two are working well, a correctly operating 
(non-caching) proxy should be pretty easy. A caching proxy, of course, 
is a whole other project.

James