[Twisted-Python] HTTP client should be more tolerant?

[Markus Stenberg]

As initial disclaimer, PLEASE change your FAQ about bug reporting - nobody
really wants to correctly report bug by first joining on some random
mailing list (OR allow moderated non-member postings, but we all know how
much spam THAT will drag in).

Anyway..

According to the HTTP RFCs, CRLF is the valid line termination
sequence. However, the appendix (section 19.3 in old HTTP/1.1, too lazy to
dig up more recent one) states as follows:

   The line terminator for message-header fields is the sequence CRLF.
   However, we recommend that applications, when parsing such headers,
   recognize a single LF as a line terminator and ignore the leading CR.

Some UNIXish servers, while breaking RFC, seem to work on browsers. I'd say
following this advice would be good, and therefore tuned my twisted http
client to be 'tolerant' (all major and minor browsers that I am aware of
are).

Changes required? web/http.py:

@@ -314,6 +314,7 @@
     length = None
     firstLine = 1
     __buffer = ''
+    delimiter = '\n'
 
     def sendCommand(self, command, path):
         self.transport.write('%s %s HTTP/1.0\r\n' % (command, path))
@@ -325,6 +326,7 @@
         self.transport.write('\r\n')
 
     def lineReceived(self, line):
+        if line and line[-1] == '\r': line = line[:-1]
         if self.firstLine:
             self.firstLine = 0
             l = line.split(None, 2)

I'm also considering implementing HTTP/1.1 client. Has anyone done anything
about it? Thoughts? 'Give up, bad idea'?

I also rewrote the proxy for my local use a bit, mostly to make it more
component-oriented AND correct (currently HTTP/1.1 clients do not get
persistent service, which is inefficient). Unfortunately changes are mixed
among some other code, but has there been any work on proxy recently? 
I might have interest in submitting patch or two to make it more correct in
some things.

-Markus

-- 
A: "You can do wonderful things in software if you get rid of the
    assumption that Windows == Shit."
B: "But likewise you could do wonderful things in hardware if you
    got rid of the assumption that Gravity == 9.81 metres per second
    squared."
C: "But that's an invalid comparison.  You can get rid of the gravity
    problem quite easily by a change of location -- but do you
    think Windows would get any better if you shot it to the moon?"
-- rec.humor.funny post by ermel at gmx.de (Erik Meltzer)