[Twisted-Python] blocking question

[Bob Ippolito]

On Thursday, Apr 17, 2003, at 23:43 America/New_York, Andrew Bennetts 
wrote:

> On Thu, Apr 17, 2003 at 11:15:06PM -0400, Konrad Rokicki wrote:
>> Hey everyone, newbie question:
>> Are there any Python/Twisted idioms for doing blocking? More 
>> specifically,
>> in a client I have a method to send a query to the server and another
>> method that gets called when the query is answered. Is there an easy 
>> way
>> to say "send the query then wait x seconds for it to return in this
>> function"?
>> It seems like blocking is taboo in twisted and if anyone has an
>> alternative I'd be glad to hear it. I guess I could do this
>> asynchronically but it would be a pain I think.
>
> See http://twistedmatrix.com/documents/howto/defer for more details on 
> using
> Deferreds.  See also the "Writing a Server" and "Writing a Client" 
> howtos.
> This is pretty basic Twisted stuff, so the existing docs cover it 
> pretty
> well.

Is Glyph's Deferred talk from PyCon online yet?  That might also be 
useful.

krad - You should've just gone to PyCon man, it was right in DC at GWU 
and there were god knows how many talks about various pieces of Twisted.

The only time threads are used in Twisted is if an API that only 
provides a blocking interface must be used.. for example, 
twisted.enterprise uses a threadpool to dispatch queries to DBAPI 
compliant databases, since DBAPI is synchronous and blocking.  The only 
other way around this is to have a pool of worker processes instead of 
worker threads, but you don't really have anything to gain with this 
approach b/c it uses more memory/resources and IPC is slower than 
thread communication.

But yeah, what you want to do is definitely the kind of thing that you 
should be using Deferreds for.. Also, due to the chaining behavior, you 
can do something practical like this:

from twisted.web import microdom, client
# Start webpage fetch
d = client.getPage('http://example.com/somewebpage.html')
# If webpage fetch takes more than aFewSeconds, raise error
d.setTimeout(aFewSeconds)
# Take the web fetch result, and parse it into a DOM tree
# equivalent to saying: microdom.parseString(webFetchResult, 
beExtremelyLenient=1)
d.addCallback(microdom.parseString, beExtremelyLenient=1)
# this will get triggered if the timeout happens, or if there's a parse 
error
# however, with beExtremelyLenient, there usually won't be one.. it can 
parse some f'ed up html
d.addErrback(yourTimeoutOrParseErrorHandler)
# If everything goes OK, or if for some reason 
yourTimeoutOrParseErrorHandler returns a non-Failure result
# call yourPageHandlerThatUnderstandsDOMTrees(result)
# because microdom.parseString was called on the result, it is now a 
DOM tree instead of a string
d.addCallback(yourPageHandlerThatUnderstandsDOMTrees)

Another convenient thing about deferreds is that if the result of some 
callback is a deferred, then it will automagically be chained as 
well... for example, if yourPageHandlerThatUnderstandsDOMTrees wants to 
pass a fetched URL to the next callback handler, then all it has to do 
is return client.getPage(someURL) and the next handler will receive the 
result of that request, not the deferred instance for that request.

-bob