[Twisted-Python] Re: nntp protocol supports moderated newsgroups?

[Barry A. Warsaw]

>>>>> "JC" == Jp Calderone <exarkun at twistedmatrix.com> writes:

    JC>   I wish I could say that is the case -- I've just been
    JC> motivated to work on other projects lately.

I can sympathize!
    
    JC> I agree that it has gotten more complex, and I don't like it.
    JC> I've recently been pointed at twisted.persisted.dirdbm, and my
    JC> current thinking is that PickleStorage and
    JC> NewsStorageAugmentation should go away in favor of a
    JC> DirDBMStorage.  PickleStorage was never meant as anything more
    JC> than a toy, and the SQL stuff doesn't seem to have yielded any
    JC> performance benefits, probably due to the complexity of some
    JC> of the common queries it is performing.

PickleStorage was fine for my purpose, which wasn't to create a fully
functional peering news server, but to have something
protocol-complete enough to test Mailman's gatewaying functionality.

    JC> The file given to --servers should contain a list of news
    JC> servers to which to connect and share messages with.  This
    JC> hasn't yet been tested thoroughly -- I have surprising
    JC> difficulty finding a news server that implemented enough of
    JC> the protocol to let this work.  I guess the "user facing"
    JC> servers are kept intentionally non-functional for some reason.

Ah, cool.  I guess --servers should simply be optional then (as should
--moderators, but I wasn't sure how to do that).  I think there are
use cases beyond testing Mailman <wink> for having a news server that
isn't a peer of other servers.  E.g. an internal mailing list mirror.

    JC> Yea, this is new too, since pickle was the only legal command
    JC> in the past.  If I go with the dirdbm stuff, I'll probably
    JC> make it the default and let pickle or sql be specified if
    JC> someone wants something different (Or pickle and sql might
    JC> quietly disappear - the prospect of maintaining 3 different
    JC> backends - 2 of which are likely useless to nearly everybody -
    JC> is a little unpleasant).

Pickles were fine for me, but I can understand others may not be happy
with them.  Sql wasn't an option because I don't have an sql server
set up, and that would have been too high a barrier for me.

>  While these repeated as long as the server was running, they didn't
>  seem to affect its operation.

    JC> I'm guessing these are due to whatever you put in your server
    JC> list file.

Makes sense!

>In any event, I hacked in moderated group support by adding a
>-m/--moderators flag, which takes a file containing a list of
>newsgroup/moderator address pairs.  When a message is POSTed to a
>group, the list of target newsgroups is checked against the
>moderators.  If there is any match, and the article has no Approved
>header, the article is not posted, but instead emailed to the
>moderators address.  The value of the Approved header is completely
>ignored -- it just has to be there.  I believe this is pretty close to
>how Usenet s/w handles moderators.

    JC> Hmm.  How does a moderator then approve the message?  The
    JC> interface I had envisioned was something closer to mailman's
    JC> scheme, though this is certainly easier on the server side :)

I believe that the news server never really enters into the picture
for moderation.  All it cares about is that posts to moderated groups
have an Approved header -- it doesn't even care what the value of that
header is!  If the Approved header exists, it accepts the post.  If
not, it forwards it to the email address of the moderators.  That's
it.

Mailman's moderation support (only in cvs at the moment) works with
this protocol so that unmoderated messages are held for approval via
the normal admindb interface.  If the list owner -- who doubles as the
newsgroup moderator -- approves the message, Mailman re-posts it
including an Approved header, to its local news server.  The news
server will accept the posting now and everything proceeds as normal.

So the news server's behavior is only half the picture (but a critical
half!).

>- What to do if the list of target newsgroups has more than one
>  moderated newsgroup in it?  Off hand, I'm not sure what Usenet does
>  in this case.
>
    JC> I believe the correct behavior would be to post the message to
    JC> all un-moderated groups and hold the message for approval from
    JC> the moderators of each moderated group, posting it to each
    JC> held group when the appropriate moderator approves it.

Except the news server can't do this, because of the above.  I /think/
what actually happens in practice (some Usenet expert will have to
verify this) is that the news server doesn't post the message to any
newsgroup, but forwards it off to the moderators of the first
moderated newsgroup in the list.  Once they approve it and repost,
that's all the approval necessary.
    
>- Any un-Twisted coding offenses. :)

    JC> The only one I see is the use of smtplib.  Its sendmail method
    JC> will very likely block, possibly for a long time, hanging the
    JC> entire server.

Ah, I'm so used to Mailman's behavior, I hadn't tought about this.
Mailman always connects to a trusted, local MTA to do the final
delivery.  Usually this is some server on localhost:25.  Mailman never
connects to the actual recipient's remote mta.  I'd be happy enough if
Twisted made the same assumption, but OTOH since Twisted /does/ come
with an SMTPd, it can probably do better.
    
    JC> Aside from just being nasty, this opens up the
    JC> possibility for DoS on servers, simply by repeatedly mailing a
    JC> moderated group.  I'm going out of town for a week this coming
    JC> tuesday, and I doubt I'll have time to figure out how
    JC> twisted.protocols.smtp.SMTPClient works before then, so I'm
    JC> going to hold off on committing this.  If you want to figure
    JC> that out and send another patch, that'd be great, otherwise
    JC> I'll just take care of it when I get back.

I probably wont' get to it.  Twisted + my hacks was enough to debug
Mailman's new moderation support, and I'll have to leave it at that
for now.  I've got my hands full trying to get Mailman 2.1b4 out the
door this weekend!

  Glad to hear it :)  Thanks for the patch.

No problem, thanks again for your help.
-Barry