[Twisted-Python] How can ftpclient.py work?

[Andrew Bennetts]

On Sat, Jul 26, 2003 at 08:28:53PM -0500, John Goerzen wrote:
> Hi,
> 
> I've been reading over the Twisted examples to learn the system and
> one of them, ftpclient.py, has me flummoxed.  Specifically, the
> connectionMade function near the end.

It's worth noting that that example is meant more as a brief example of how
to use ftp.FTPClient, rather than as a real application.

> It issues a number of commands right in a row, without waiting for the
> results to come back from the deferreds that they return.  To me this
> seems bad because:
> 
> 1. The system would have no way of knowing which command a particular
>    response belongs to

I think you're referring to these parts:

    # Get the current working directory
    ftpClient.pwd().addCallbacks(success, fail)

    ... snip ...

    # Change to the parent directory
    ftpClient.cdup().addCallbacks(success, fail)

You're right, in a real application, you probably do want to distinguish
between the responses from these two commands.  That could be as simple as
having different success functions for different commands, e.g.

    ftpClient.pwd().addCallbacks(handlePWD, fail)
    ftpClient.cdup().addCallbacks(handleCDUP, fail)

Or you could pass extra arguments to the callbacks:

    # Using .addCallback
    ftpClient.pwd().addCallback(success, 'pwd').addErrback(fail)
    # Using .addCallbacks
    ftpClient.cdup().addCallbacks(success, fail, callbackArgs=('cdup,))

In that case, the success function might be defined as:

    def success(response, command):
        print 'Success!  Got response for command %r:' % (command,)
        print '---'
        if response is None:
            print None
        else:
            print string.join(response, '\n')
        print '---'

> 2. Sending additional commands before the server is done with the
>    first one may cause problems (for instance, if it takes a little
>    while to fully transmit the first command, the second one could be
>    transmitted in the middle)

This is not a problem for FTPClient -- it internally queues the commands and
issues them when the server is ready for them.

[In the case of FTP, Twisted's FTPClient doesn't attempt any funky
pipelining, so it's simply a matter of waiting for a response to the last
command if there's a response outstanding, otherwise it can be issued
immediately.  See the implementation of FTPClient.sendNextCommand and
FTPClient.lineReceiver (which calls sendNextCommand when appropriate) for
details.]

In general, though, most protocols would simply be doing something like:

    def sendRequest(self, data):
        # ...
        self.transport.write(self.requestHeader)
        self.transport.write(data)
        # ...

And the transport takes care of making sure the data is sent in the right
order.  You're probably thinking "but what if two calls are made to
sendRequest at the same time?  Isn't there a race condition?"  The answer is
technically yes -- very little of Twisted is thread-safe[1].  But in practice,
Twisted programs don't need to use threads to service multiple connections,
so you don't have to worry about these sorts of problems.  Only one thing
is happening at any one time in Twisted, the trick is to make sure none of
those things blocks so that *looks* like it's doing several things at once.

If you grep the Twisted source, you'll see that virtually nothing uses any
thread-locking tools like threading.Lock :)

[1] But if you need to safely call a function from a thread, you can use
    reactor.callFromThread, which will schedule that function to be called
    from the main event loop.

> 3. There is no error-checking in there, so commands are issued that
>    are dependant on the success of earlier ones (rnuning nlst after
>    cdup, for instance), but they're issued potentially before the
>    earlier ones have a chance to return an error.

In the case of doc/examples/ftpclient.py, the commands Deferreds' have
'fail' added as an errback.  It's defined as:

    def fail(error):
        print 'Failed.  Error was:'
        print error
        from twisted.internet import reactor
        reactor.stop()

So there's no problems there -- the error handler will shutdown the entire
program by calling reactor.stop().  Probably not what you want to do in a
real application, but sufficient for an example.

> Now, I'm assuming that these problems do not actually exist.  Yet I
> cannot work out why not.  Can anybody shed some light?

I hope I've shed some light for you -- let us know if you need more :)

-Andrew.