[Twisted-Python] Non returning deferred result.

Fri Apr 18 00:35:50 MDT 2008

Hey everyone,

Sorry about the subject line, its a difficult problem to sum up in a
single line subject. Give me a little time to explain.

I'm using twisted.web2. I have a generic webservice over http. It
receives a file via standard POST http upload. It saves this to disk
using non blocking stream setup as outlines in web2.static.FileServer.

Now I don't immediately return a response, because another thread then
processes this file and builds an output file. So I return a deferred
from my render() function. When the output file is finished building,
the deferred callback is triggered with the http.Response() object and
the response is finally returned to the waiting http connection.

This DOES work except for a little strange behavoir. When the deferred
callback is triggered, the client doesn't see any response, the
connection just stays open. Then if I connect to the webservices from
anywhere, even a bad URL, I get my response suddenly back.

It's as if the callback is not triggered until some TCP activity cause
the main reactor to pump the pending requests or something. Is there
anyway to get this deferred callback to be recognised and processed
immediately? I could do a kind of hack, where the deferred callback is
called, and then a quick connection is made to the webservice to "pump"
the message through, but it doesn't seem very 'right' to me.

Heres some summary of the approach incase the devil is in the details...

My Resource class:
==================
class UploadFile(resource.PostableResource):
	# 1 gig is max file upload size
	maxSize=1 * 1024 * 1024 * 1024

	def render(self, ctx):
		"""Create the job for this upload and return the relevent deferred"""
		request = iweb.IRequest(ctx)

		job=Job()
		deferred = job.SubmitInput(request.files)				

		return deferred

My deferred callback:
=====================
deferred.callback( http.Response(responsecode.OK,
				{'content-type': http_headers.MimeType('text', 'html')},
				"Success!" ) )

My TopLevel:
============
class Toplevel(resource.PostableResource):
	addSlash = True
	def render(self, ctx):
		return http.Response(responsecode.OK,
				{'content-type':
						http_headers.MimeType('text', 'html')},
						"""
<html>
    <form action="http://localhost:8080/uploadfile"
            enctype="multipart/form-data"
            method="post">
        filename: <input type="file" name="filename">
		filename2: <input type="file" name="filename2">
        <input type="submit" value="submit">
    </form>
</html>
""")
	child_upload = UploadFile(jobqueue)

My server startup:
==================
# Create the resource we will be serving
test = Toplevel()

# Setup default common access logging
res = log.LogWrapperResource(test)
log.DefaultCommonAccessLoggingObserver().start()

# Create the site and application objects
site = server.Site(res)
application = service.Application("demo")

# Serve it via standard HTTP on port 8080
s = strports.service('tcp:8080', channel.HTTPFactory(site))
s.setServiceParent(application)

Im running the application as:

twistd -noy server.py

Twisted v8.0.1+, actually, the lastest SVN head.

Any one with more knowledge of twisted's internals have any ideas on
what is happening? Does anyone have any good ideas on how to make the
deferred callback trigger an immediate response (without opening a new
TCP connection to kick it into gear)?

Thanks

Regards
Crispin Wellington
<cwellington at ccg.murdoch.edu.au>