[Twisted-Python] Re: Considering Twisted for OfflineIMAP

[John Goerzen]

Andrew Bennetts <andrew-twisted at puzzling.org> writes:

> Ooh, nifty :)

Thanks :-)
>
> Twisted uses non-blocking sockets, so it can do an arbitrary amount of
> connections (both client and server) without any threads.  Running multiple
> clients (or servers) at a time is as easy as calling reactor.connectTCP more
> than once.

Well, yes and no, I think.  Let me elaborate.

OfflineIMAP always behaves as an IMAP client, so I'll be using
twisted.protocols.imap4.IMAP4Client.  This class is written to use one
and exactly one connection to the server (it must, since it takes a
single SSL contextFactory as an arg to __init__).

OfflineIMAP would like to do, say, 4 operations at once.  For the sake
of example, let's say these four are:

1. Getting a list of messages in folder A
2. Downloading a message from folder B
3. Uploading a message to folder B
4. Deleting a message in folder C

Now, the way I'd handle this pre-twisted is that my own IMAP folder
class would basically do this:

imap = pool.acquireconnection()
try:
    # do the stuff here
    # imap.select() to pick the relevant folder
    # imap.download() or whatnot
finally:
    pool.releaseconnection(imap)

This was running with multiple threads, so even though the stuff in
the "try" was blocking, it didn't matter. 

With Twisted, I could clone this setup somewhat by making sure
the last callback and errback in my deferreds from IMAP4Client 
release the connection, that's ugly and error-prone.

I'd imagine that downloading a message would basically go like this:

d = pool.acquireconnection()
d.addCallback(select)
d.addCallback(download)
d.addCallback(releaseconnection)
d.addCallback(returnresults)

But this doesn't take into account errors, and it rather annoying to boot.

So my question is: just how might I accomplish this in a nice way with
Twisted?

> [What happened to question 2?]

I found the answer to it before I posted the message, so I deleted it
but forgot to renumber.

>> (e.g. twisted.internet.utils.getProcessOutput), a function call in another
> thread completing (e.g. twisted.internet.threads.deferToThread), and so on.

Got it.

>> 4. The howto "book" alludes to pending improvements on the mail
>>    infrastructure.  Anything I should be aware of here?
>
> Jp has been doing a lot of great work on this recently -- you should
> definitely be working with CVS rather than 1.0.6 if you can.  I don't know
> much about the details, though.

Do the online API docs refer to the CVS version or 1.0.6?  It's 1.0.6 that
I've got right now.

> I suspect this can be dealt with, but I'm not sure off the top of my head
> how... hopefully someone else can help you here.

One thought I had was to wait to import the reactor until I know what
kind of UI I want.  That works OK as long as I continue using
ConfigParser for my configuration.  If at some point I switch to using
Twisted's config storage (I know it's there but I haven't looked at it
yet), I suspect that could introduce a chicken-and-egg problem.

-- John