[Twisted-Python] protocol flows

[Clark C. Evans]

Ok.  Flow has gone through a rather largish growth 
period in the last week.  Improvments include:

  1.  It now supports protocols, ie, you can use generators
      to build protocols.   This is not complete, nor is it
      documented well, but that will come in time.

      Currently it only supports protocols with one callback, 
      but the flow.py module is quite close to supporting
      N callbacks via flow.Concurrent (which is still being
      tested and refined).

      The flow support for protocols still has some issues
      hiding errors.  I've done quite a bit to make exceptions
      visible... but I'm still a twisted newbie, and a few are
      slipping through the cracks (or rather hiding between
      the cracks).   The process of making user-level errors
      clearly reported is on-going.  

  2.  It has quite a few speed improvments, overall flow isn't
      fast beacuse it basically implements cooperative 
      multitasking.  Generally, given noop functions, flow
      is about 5-7x slower than direct generators.  However,
      most items are not noop, so the actual factor in practice
      is more like 20% to 2x slower.   Not a bad price for 
      relatively easy to read code.

      I'm not sure how many optimizations are possible; one 
      recent optimization speeded up cases whith lots of little
      results (but slowed larger more infrequent results).  This
      optimization was due to exarkun's suggestion.   Anyway, 
      this change I'm certain has introduced bugs, but they will
      be squashed in time.

      Of course, writing the whole module in "C" would give a 
      big bonus.   However, flow must reach serious stability
      and a large user base before that is taken on.  I think
      that the speedup from C coding would be quite significant,
      but not earth-shattering.

  3.  A new Callback stage was added.  This was needed for 
      protocol support, but can be useful standalone.   Other
      stages include a List and String stage.  Not that those
      are horribly useful, but they do help to illustrate 
      what is going on, so I figured they'd be nice to have.

  4.  Some serious refactorings.  Most importantly in this one
      is a CallLater instruction.  flow.Threaded is much faster
      with this change.   If you were following the source, I'd
      start from the top down again.   

A few things to note:

  A.  Some of the interface has changed for those not using
      for-each.   In particular, the state functions (isFailure, etc.)
      are gone as they were horribly slow.  Furthermore, stage.result
      is now stage.results.   

      If you were using next() explicitly or implicitly via the
      for-loop, then you will have no problems; as this aspect of
      the interface hasn't changed.

  B.  This module is approaching stability.  I don't expect many 
      more changes... so, if you have comments.  Please... let's
      hear them.

  C.  I don't know if/when flow will be added to twisted, so
      it's still a use-at-yer-risk

Short-term plans:

  I.   Get the protocol to support multiple callbacks

  II.  Add more regression tests, especially ones with
       icky user-level failures to make sure that errors
       are being propigated.

  III. Start writing a few 'flow-filters' and such that work
       with flow protocols.  For instance, one filter could
       be the equivalent of LineReceiver.  Perhaps refactor
       common code into a separate module...

  IV.  Update/improve the documentation, flow.html


Here is an echo "flowprotocol":

        from __future__ import generators
        import flow
        from twisted.internet import protocol, reactor
        PORT = 8392
        
        print "client                   server"
        print "-------------------      ---------------------"
        def echoServer(conn):
            print "                         connected, now wait for client"
            yield conn
            print "                         client responded"
            for data in conn:
                print "                         received '%s', sending back" %
        data
                yield data
                print "                         waiting for client again"
                yield conn
            print "                         disconnected"
        
        server = protocol.ServerFactory()
        server.protocol = flow.makeProtocol(echoServer)
        reactor.listenTCP(PORT,server)
        
        def echoClient(conn):
            print "connect and send"
            yield "Hello World"    # sending data
            print "waiting for server"
            yield conn
            print "received '%s'" % conn.next()
            print "sending more"
            yield "Another Line"
            print "waiting for server"
            yield conn
            print "received '%s'" % conn.next()
            reactor.callLater(0,reactor.stop)
            print "disconnecting"
        
        client = protocol.ClientFactory()
        client.protocol = flow.makeProtocol(echoClient)
        reactor.connectTCP("localhost", PORT, client)
        
        reactor.run()

Which produces...
        
        client                   server
        -------------------      ---------------------
        connect and send
        waiting for server
                                 connected, now wait for client
                                 client responded
                                 received 'Hello World', sending back
                                 waiting for client again
        received 'Hello World'
        sending more
        waiting for server
                                 received 'Another Line', sending back
                                 waiting for client again
        received 'Another Line'
        disconnecting
                                 disconnected
        

That's it.

Clark