[Twisted-Python] Interactive Twisted client (newbie question?)

[Jp Calderone]

On Wed,  5 Jan 2005 15:44:33 +0000, Jon Blower <jdb at mail.nerc-essc.ac.uk> wrote:
>Hi Glyph,
> 
> Thanks very much for your reply, very useful.  I've had another think about my
> problem and I guess it boils down to this question: when I'm writing a client
> program, how can I simulate a synchronous function call (avoiding callbacks)
> like this pseudo-Python, which sends a message then returns its reply:
> 
> def sendMessage(message):
>    transport.send(message)
>    # Wait for reply to arrive
>    reply = getReplySomehow()
>    return reply

  One way is to have getReplySomehow() context switch out of sendMessage() 
and into something capable of constructing getReplySomehow()'s return 
value, then switching back and returning it to be bound to the name reply.

  Twisted has ignore this rather successfully up until recently.  It is,
after all, a basic language feature, and should be provided by the runtime.
Nothing about it is Twisted specific, and it is perfectly possible to write
a Twisted program without this trick.

  Recently Chris Armstrong's "deferred generator" support was added to 
t.i.defer, though.  With it, you write code something like this:

    from twisted.internet import defer

    def sendMessage(message):
        transport.send(message)
        intermediate = defer.waitForDeferred(getReplySomehow())
        yield intermediate
        reply = intermediate.getResult()
        yield reply
    sendMessage = deferredGenerator(sendMessage)

  This supposes that getReplySomehow() returns a Deferred, and causes
sendMessage to also return a Deferred (one which fires with reply 
eventually).  The benefit is not immediately obvious here, since you 
are handing reply back to your caller, rather than using it.  The exact
equivalent code without this technique is:

    def sendMessage(message):
        transport.send(message)
        return getReplySomehow()

  However, there is some benefit if, instead of returning reply, sendMessage
uses it locally a bit.

  There are other solutions, of course.  You can use pre-emptive threads,
as provided by Python's threading module (or Twisted's thin layer on top
of it).  Keep in mind that there is pretty much only one threadsafe API in
Twisted (callInThread).  If you do this, you will have to be extremely 
careful not to call Twisted APIs from the wrong place.  There is an example 
of a somewhat generic implementation of this solution in my sandbox.  
It gives the appearance of working (and may even do so), but consider it 
untested and loaded with bugs (I'm certainly not using it in any of my 
software).

  Greenlets are another solution along these lines.  They simply replace
pre-emptive kernel threads with cooperative user-space threads.  As such,
you do not need to worry about calling Twisted APIs from the wrong thread,
since there is no wrong thread.  Greenlets are implemented with some tricky
voodoo, so once again, while I know of no bugs, I also wouldn't be surprised
if using them caused a rift in space/time to open up and eat your head.

  So what's a Python programmer to do?  Well, the technique I'll actually 
recommend is the simplest: callbacks.  Oh, they're a pain to write sometimes,
but they're much less magic than the other solutions described, often perform
better, and all the bugs you'll get when using them are the simple to fix kind,
rather than the my-stack-just-got-swizzled-time-to-segv or the 
oh-there's-a-race-that-only-triggers-under-load-on-someone-else's-machine kind.
They also lead to rather more modular code than the synchronous-style equivalents.
You can unit test a callback by itself.  How do you unit test lines 7 - 11 of a
single function?

  As to your wondering about Twisted's utility for clients versus servers, I 
think you are close to the truth.  The actual divide, though, is concurrency.
If your program only needs to do one thing at a time, Twisted doesn't simplify
much.  If your program needs to do two or more things at once, then it does.
Keep in mind that, even for a client, there are often at least two things that
need to be happening: talking to the network and updating the user interface.

  Hope this helps,

  Jp