[Twisted-Python] Patch for twisted.protocols.nntp

[Matthias Urlichs]

Hi, Jp Calderone wrote:

>  They can complete in any order because the code responsible for the
> Deferred's callback must buffer the message in memory.
> 
Nope. Converting the code to assemble the articles in file-like objects,
or whatever, instead, wouldn't make much of a difference to my current
code -- they would still be able to complete in any order.  Part of the
niceness of streaming CHECK and TAKETHIS messages to an NNTP server is
that there is no ordering required; you can't do that with SMTP.

On the other hand, thinking about this I just had an idea how to manage
the transition from a bunch of parallel Deferreds to a serialized
producer/consumer scheme (addCallback() a code snipped which enqueues the
Deferred onto a queue, and take it from there one-by-one). It's still
going to start out as a Deferred, though; "a producer which needs to do
something special when it first delivers data and which might trigger an
error instead" is complicated code which lumps at least three concepts
into one piece of code, while using a Deferred in the first step leads to
nicely separated stages with distinct code -- always a win in my book.

> At the very least, the object that is currently a string should become a
> file object (This should have minimal impact on your code -- if
> buffering in memory is acceptable to you, you only have to build a
> StringIO object from your string)

That would be no problem. Whether the Deferred returns an in-memory string
or a file object doesn't make a whole lot of difference, esp. as a simple
'if not hasattr(f,"read"): f = StringIO.StringIO(f)' is sufficient for
turning one into the other.

> It -does- seem that this is becoming a common task (smtp.py and pop3.py
> do it, imap4.py needs to do, and now nntp.py), so maybe I'll write a
> utility mixin to help out, and I'll see if I can generalize it away from
> just file-like objects too.

That would be nice to have. I'd write it as an adapter or similar, i.e. a
class which turns an LF-based dots-ignorant possibly-without-final-LF
producer into one which has CRLFs, dot-escapes, and a guaranteed CRLF at
the end. That way it'd be useable for things which aren't a streaming
protocol. Direct back-end access to the NNTP storage comes to mind;
it stores articles in the latter format.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf at smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
-- 
There will always be some delightful mysteries in your life.