[Twisted-Python] Hep and Twisted (Was: Twisted-webserver hangs when serving big files)

[Abe Fettig]

On Wed, 2003-06-11 at 06:27, Jp Calderone wrote:
> On Mon, Jun 09, 2003 at 01:35:56PM -0400, Abe Fettig wrote:
> > [snip]
> > 
> > * The SMTP server: Currently going through a rewrite in CVS.  I need to
> > set up code to manage the outgoing message queue, and keep track of
> > which messages have been delivered to which locations.  Is there code in
> > twisted.mail to do this? (Keep in mind some messages are being delivered
> > to blogs and other non-SMTP locations)
> > 
> 
>   There will be code for this in the relatively near the future -- this is a
> subtask of an item high on my todo list.
> 
> I'll check out your blog and maybe some of the Hep source (if it is
> appropriate to do so?) to get a better idea of your use case, so I can keep
> it in mind.
> 
>   If you want to toss any info my way regarding the details of your use
> case, I'll keep them in mind as I work on the implementation.

I'd like to be able to inherit from a class that handled all the queue
management bits (keeping track of which messages are in the queue,
calling back with a notification of which messages were and were not
delivered, etc).  Something like this:

Class MyQueue(MessageQueue):

	def deliverMessage(self, message, destinationAddress):
		[ my delivery code ]
                [ return a deferred ]
		

q = MyQueue()
d = q.addMessageToQueue(myMessage, destAddresses)
d.addCallback(allMessagesWereDelivered)
d.addErrback(someMessagesWereNotDelivered)


This doesn't take into account cases where you're delivering the same
message to several users on the same SMTP server (a very common case if
you're simply using an upstream server for all delivery).  In this case
calling deliverMessage() once for each address is bad.  Maybe it would
be better if the inheriting class could specify how different groups of
addresses should be delivered, like this:

def getDeliveryFunctionsForAddresess(self, addresses):
	deliveryFunctions = {
		self.deliverToLocalServer:[], 
		self.deliverToUpstreamServer: []}

	for address in addresses:
		if address.endswith("@mylocaldomain")
			deliveryFunctions[self.deliverToLocalServer].append(address)		
		else:
			deliveryFunctions[self.deliverToUpstreamServer].append(address)

	return deliveryFunctions

Another problem is that the queue should be maintained between
sessions.  The server may be shut down while a message is still waiting
to be delivered to one or more destinations.  In that case you may not
be able to rely on getting a callback when a message is delivered (since
the original deferred has probably been lost). Maybe it would be better
to register a function with the queue that gets called each time a
message is delivered, or fails?

q = MyQueue
q.notifierFunction = notifyUserOfDelivery

Anyhow, I'm sure you'll find a better solution than the ones I've listed
here. But hopefully that gives you an idea of what I'd be looking for -
the important thing is that I be able to change the delivery behavior
based on the destination address. If you'd like to take a look at the
code in Hep 0.3.3 you're welcome to - see servers/smtp.py and
agents/deliver.py.  Just keep in mind that this code is 6 months old,
and the architecture of Hep has changed somewhat since then.

Hep can easily convert messages to and from email.Message and/or rfc822
text objects, so I don't have a problem with your code depending on one
of those formats.

Abe