[Twisted-Python] Considering Twisted for OfflineIMAP

[Andrew Bennetts]

On Sat, Jul 26, 2003 at 08:48:15AM -0500, John Goerzen wrote:
> Hello,
> 
> I am the author of OfflineIMAP, a multi-threaded multi-account
> bi-directional synchronization tool
> (http://quux.org/devel/offlineimap) written in Python.

Ooh, nifty :)

[...]
> 
> My questions, then, are these:
> 
> 1. How does Twisted interact with things like threads and connection
>    pooling?
>    
>    I currently let the user control the maximum number of connections
>    that are open to a given server, and whenever the code needs to
>    access an IMAP server, it grabs a connection from the appropriate
>    pool and uses it (using threading primitives to block if no
>    connection is available.)  Can Twisted handle multiple connections
>    to multiple servers doing multiple things at once?  Can it do
>    connection pools?  If it needs threading to do these things, does
>    it play nice with it?

Twisted uses non-blocking sockets, so it can do an arbitrary amount of
connections (both client and server) without any threads.  Running multiple
clients (or servers) at a time is as easy as calling reactor.connectTCP more
than once.

[What happened to question 2?]

> 3. I notice the "Deferred" object and the benefits it provides when a
>    server response is expected to take some time.  I assume that some
>    sort of internal threading is taking place here?

No threading.  Deferreds aren't magical at all.  Deferreds are simply an
object a function can return to say "I don't have a result ready yet; as
soon as I've got it, I'll let you know."  The caller can then register
callbacks and errbacks that will be called when the result does become
available.

How the Deferred eventually gets called is up to the code that originally
created it; it might be the result of query sent to the network being
answered (e.g. twisted.web.client.getPage), a spawned process terminating
(e.g. twisted.internet.utils.getProcessOutput), a function call in another
thread completing (e.g. twisted.internet.threads.deferToThread), and so on.

>    What if the server *request* takes some time -- say, uploading a
>    2MB e-mail over a dialup link.  Are we still OK with handling that
>    in a non-blocking fashion?

Absolutely!  Twisted handles large transfers via a consumer/producer
interface, though, rather than with Deferreds (which are one-shot results,
rather than streaming).

> 4. The howto "book" alludes to pending improvements on the mail
>    infrastructure.  Anything I should be aware of here?

Jp has been doing a lot of great work on this recently -- you should
definitely be working with CVS rather than 1.0.6 if you can.  I don't know
much about the details, though.

> 5. Is there any sort of unified mail API in Twisted (like JavaMail)
>    that would present me with a single API to both IMAP and Maildir
>    repositories, or is that something I need to do on my own?  (I've
>    already done it, so it's no big deal to do that again)

I've no idea -- Jp?

> 6. OfflineIMAP supports several different user interfaces (two written
>    with Tkinter, 1 using Curses, and three plain console ones.)  I would
>    like to reimplement the Tkinter ones with wxPython, and am glad to
>    see that Twisted works well with this.
> 
>    One concern, though, is that the users can supply a list of UIs to try
>    in the config file: for instance, Tk, Curses, Noninteractive.  The
>    system will try each one in turn until it reaches one that works.
>    (For instance, Tk may not work if the user is not in X, and Curses
>    may not work if stdout is not a tty).  It looks like this may be
>    problematic with Twisted since the UI selection always seems to be
>    known in advance with the import commands related to the reactor.
>    Any thoughts here?

I suspect this can be dealt with, but I'm not sure off the top of my head
how... hopefully someone else can help you here.

-Andrew.