[Twisted-web] Deferred fun

[Andrew Bennetts]

On Mon, Jan 16, 2006 at 07:56:14PM -0600, James R. Saker Jr. wrote:
> Per new years resolution 'I will learn how to use deferreds in twisted', I've 
> got a little puzzle in an xmlrpc remote nmap tool that I put together to try 
> to learn deferreds (as well as automate scans on my firewall from outside my 
> net), figuring the speed of nmap returning the results of a scan can be 
> excessively long and should make for a good deferred experiment. Except I 
> keep hitting an AssertionError and think it may have to do with what happens 
> to my deferred when it completes:

I'll point out the problems that jumped out at me:

> #!/usr/bin/python
> """
> pyrscand.py
> Python remote scanner: Nmap XML-RPC module for processing scan requests from a 
> remote host
> """
> import os
> from twisted.internet import defer
> from twisted.web import xmlrpc, server
> 
> NMAP_PATH = '/usr/bin/nmap' # location of nmap binary on host
> 
> class ScanServer(xmlrpc.XMLRPC):
>     """
>     XMLRPC server for remote scan processing
>     """
> 
>     def handleScan(self, type, ip, detectOS, timing):
> 	"""
> 	Simplistic scans only for now!
> 	"""
>         result = ''
>         print "running scanner..."
>         scandata = os.popen('%s -%s -A -T%s %s -oX -' % \
>             (NMAP_PATH, type, timing, ip), 'r')

os.popen is not safe to use with Twisted, due to the joys of signal handling.
See the Using Processes developer guide for details:
http://twistedmatrix.com/projects/core/documentation/howto/process.html

The short answer in this case is to use twisted.internet.utils.getProcessOutput:

    def handleScan(self, type, ip, detectOS, timing):
        return getProcessOutput(NMAP_PATH, 
            args=['-' + type, '-A', '-T' + timing, ip, '-oX', '-'])

(docstring omitted for brevity)

>     def handleFailure(self, f):
>         print "errback"
>         print "we got an exception: %s" % (f.getTraceback(),)
>         f.trap(RuntimeError)
>         return 'Error'

I'm not sure why you're catching RuntimeError, as it's never raised anywhere
that I can see.

>     def xmlrpc_scan(self, type, ip, detectOS, timing):
>         """
>         Inputs: type (sS, sT, sP), ip, detectOS flag, timing performance (0-5)
>         Outputs: XML formatted report value compliant with nmap schema
>         """
>         print "scanner"
>         d = defer.Deferred()
>         d.addCallback(self.handleScan(type, ip, detectOS, timing))

In general, when you want a callback to be run with arguments, you say:

    d.addCallback(func, arg1, arg2)

*not*:

    d.addCallback(func(arg1, arg2))

Because the latter will call the function immediately, rather than make it all
callback.

Also, a callback function needs to expect the result of the deferred as its
first argument, before any extras you pass to addCallback.

But this is all moot, because what you really want is for handleScan to *create*
and *return* a Deferred, not be a callback, because it is handleScan that is the
operation that you are waiting for the result from, rather than being an action
you want to take when you get a result.  Helpfully, getProcessOutput returns a
deferred of the process output for you, so your xmlrpc_scan function can become:

    def xmlrpc_scan(self, type, ip, detectOS, timing):
        d = self.handleScan(type, ip, detectOS, timing)
        d.addErrback(self.handleFailure)
        return d

-Andrew.