[Twisted-web] HTTP client TODOs

[Scott Lamb]

On Nov 4, 2005, at 8:39 AM, dreid at dreid.org wrote:
> Ok, time to clear things up about the web2-client.  Do not count on
> branches/dreid/web2-client/ It isn't going to be maintained.  Why?  
> Because
> it is full of kludges, doesn't even do http/1.1 properly and is  
> known to
> break intermittently.

It doesn't do HTTP/1.1 properly? What's wrong?

>   dreid/web2-client was created for my employer, it
> works in the confines of what we're doing, which isn't very much  
> http, but
> we still needed something that worked differently than  
> twisted.web.client.

Likewise.

> Scott,
>   I feel iffy about that fix, but I've always felt iffy about the  
> way it
> creates a new request object.  I'll let foom comment on the changes  
> made
> to stream.

The POST stream fix? Yeah, I know that one's crap. But it works for  
me. My patch includes a hyperlink to the discussion of a better fix,  
but I haven't had time to actually do it.

>   And the AlreadyCalled exception is a known bug, we've
> encountered it maybe twice at work, in the 3 or so months that we've
> been using the branch of the branch.  I haven't been able to reproduce
> it reliably enough to fix it.

For a while, I had it happening consistently. Maybe I can get that to  
happen again...

> So yes, dreid/web2-client ... not my best work, it's a
> lot of kludges I'm sorry for inflicting this upon you, I'm even more
> sorry that I have to tell you that it's not going to go anywhere, and
> backwards compatibility isn't going to happen.

Dude, you can apologize all you want, but I'm calling it a tremendous  
success. Even if all copies of dreid/web2-client-related code were to  
be annihilated tomorrow, it totally saved my ass.

Let me tell you how I used it. I had to load test our HTTP-based  
protocol. I had a Java-based load tester tool that a contractor had  
spent...stories vary...one guy said four months, another said six to  
nine months...writing. He would tell them things like "if I just make  
this change, it will compile". Ridiculously low standards. Bizarre  
verbose code with no point. I had this thing dumped on me and needed  
it working in a week. It took me a couple days to figure out that  
there was nothing salvagable there at all.

So I looked for something else. I like Python. I'd heard of Twisted  
but never used it before. I saw that twisted.web.client didn't do  
Keep-Alives, which I needed. Google showed me one of your commits on  
this branch, and I started coding. In a day, I had something usable.  
It wasn't great, but it was a hell of a lot better than using the  
contractor's code. I developed it as I needed it. Now I've got a tool  
that can generate a load that is quite configurable, is reasonably  
fast, is pleasant to run (nice optparse stuff; SIGINT, SIGHUP, and  
SIGTERM do a nice graceful shutdown), reports quick results during  
the run, and generates full reports into a relational database which  
I have a web interface for and a nice graph tool. The load tester is  
700 lines of code, with verbose comments.

I have absolutely no idea how I could have done the rapid development  
I needed had dreid/web2-client not existed. Even if I have to  
completely rewrite my tool from scratch, I needed it _right_then_.  
Even if I hadn't, it was useful as a prototype.

Anyway, I don't think the situation with this branch is nearly as bad  
as you say. Yes, it's kludgey. Yes, I found like 10 bugs. But it  
essentially works, and I don't think the design is fundamentally  
flawed. Tests can be added, and it can be refactored. Other people  
like me will come along, desperately need it, and fix a bunch of  
problems.

-- 
Scott Lamb <http://www.slamb.org/>