[Twisted-Python] Synchronization techniques

[glyph at divmod.com]

On 04:35 pm, daniel at keystonewood.com wrote:
>On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
>>On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
>>>>twisted.internet.defer.DeferredLock and some of the related classes
>>>>are
>>>>what you ought to be using.
>>>
>>>Unfortunately that only gets me half way there. DeferredLock.acquire
>>>() returns a deferred. How do I return the result of a deferred from
>>>a PB remote_xxx() function?
>>
>>Just return the Deferred from the remote_xxx() function.
>
>Thanks, I didn't know I could return a deferred from a PB remote_xxx () 
>method. That detail doesn't seem to be documented in the  Perspective 
>Broker documentation, which I have read quite a few  times.

The PB documentation is not too great.  Perhaps this paper would be 
helpful to you, if you haven't seen it:

http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html#auto7

    """
    In addition, the remote method can itself return a Deferred instead 
of an
    actual return value. When that Deferreds fires, the data given to the
    callback will be serialized and returned to the original caller.
    """
>Maybe this could be highlighted in the "Complete Example" [0]  section 
>of the PB usage documentation? The examples use the  TwistedQuotes 
>application, and the IQuoter.getQuote() method always  returns a string 
>(at least I couldn't find any implementations that  return a deferred).

Please feel free to write some patches for the documentation, or open a 
doc bug describing this issue in more detail.  It's definitely an under- 
documented feature of PB.
>However, that would require rewriting most if not  all implementations 
>of IQuoter to return deferred's and/or the code  that calls 
>IQuoter.getQuote(), which demonstrates the viral nature of  twisted 
>when used in conjunction with other libraries.

I don't think that would really be the terrible burden that you suggest, 
considering the relatively small amount of tutorial documentation that 
implements or calls IQuoter.  One could also propose a separate 
interface, IDeferredQuoter, to make the distinction clearer.
>So anyway, I rewrote my server-side library to do it the twisted way 
>and return deferred's instead trying rig up some way of waiting for 
>them. I still think it would be super useful to be able to pseudo- 
>block on a deferred (i.e. allow the reactor to process other events 
>while waiting for the deferred). It is very annoying to have to 
>rewrite many layers of code when twisted is introduced into a  program. 
>I did find gthreadless.py, and maybe that would do it.  Unfortunately 
>discussion on that seems to have been dropped some time  ago...

I'm afraid that the feature you want doesn't make any sense and is, in a 
broad sense, impossible.  There are some things like it which might be 
possible - for example, http://twistedmatrix.com/trac/ticket/2545 - but 
the reactor is not reentrant and in some sense could not be made 
reentrant.

Consider this innocuous looking block of code:

    from twisted.internet.protocol import Protocol
    from make_believe import magicallyBlockOn

    class MagicalProtocol(Protocol):
        def dataReceived(self, data):
            commands = (self.buf + data).split()
            self.buf = commands[-1]
            for command in commands[:-1]:
                if command == 'QUIT':
                    self.transport.loseConnection()
                    return
                else:
                    # Deferreds are hard, let's go shopping
                    page = 
magicallyBlockOn(getPage("http://example.com/%s" %
                                                    (command,)))
                    self.transport.write("SIZE:"+len(page))

If you were using Deferreds to track the result of the 'getPage' 
operation, you could cancel the callbacks that write to the transport in 
connectionLost.  However, with magical blocking, one dataReceived method 
might be interleaved with another.  That means that every time through 
the loop, you have to check to see if the transport has already been 
disconnected - the code as presented here is buggy and will spuriously 
fail depending on the order of the connection being lost and the remote 
page being retrieved.

In this example I've been careful to accumulate all the buffer- 
management and parsing logic at the top of the method, before any 
potential re-entrancy can happen, but other code (most everything in 
Twisted's existing protocol implementations, not to mention just about 
all application code) would not be so lucky.

It might be quite feasible to implement a microthreaded runtime 
environment that lived on _top_ of Twisted and explicitly accounted for 
issues like these, but that would not really be substantially different 
than 2.5+inlineCallbacks.
>For the record, I've included updated versions of the previously 
>posted code below. I'd be happy if someone pointed out if I'm doing 
>anything wrong (with respect to twisted) in this code.

Nothing immediately jumps out at me.  I've had to write similar code in 
the past, though, and when I've had to do that, an explicit state 
machine for the state of the subprocess (or whatever asynchronous 
resource must be acquired) has been easier to deal with than a lock- 
oriented approach to it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070404/a5152cc2/attachment.html>