[Twisted-Python] deferToThread and reactor loop
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Thu Nov 1 16:03:31 MDT 2012
On 29 Oct, 05:58 pm, tobias.oberstein at tavendo.de wrote:
>Hi,
>
>I've got a question regarding
>
>deferToThread / runWithConnection
>
>I have a network server that accepts RPCs and forwards those to a
>relational database calling stored procedures.
>
>The call of the stored procedures is happening via "runWithConnection"
>.. that is on a background thread pool (since that DBs driver is
>blocking).
>
>Now, when there is a large number of RPCs coming in roughly at once,
>the latency of each call forwarded to the RDBMs increased into the 10's
>of ms.
>
>I have created a self-contained test program without DB etc that I
>think reproduces the "issue":
>
>https://github.com/oberstet/scratchbox/blob/master/python/twisted/threadpoollatency/test.py
>
>**************
>Am I correct that "deferToThread" does not immediately forward the call
>to a background thread, but only the next time the reactor loop runs?
>**************
Hi Tobias,
Sorry, I haven't had a chance to read your minimal example in any
detail. However, I can direct you to the implementation of
deferToThread:
http://twistedmatrix.com/trac/browser/trunk/twisted/python/threadpool.py#L119
Notice the `self.q.put(o)`. This matches up with the call to
`self.q.get` in the same module:
http://twistedmatrix.com/trac/browser/trunk/twisted/python/threadpool.py#L158
Together, these bits of source should demonstrate that there's no
waiting for a reactor iteration before the work is enqueued. The work
goes into the Queue instance, and instantly any worker thread is free to
grab it.
Jean-Paul
More information about the Twisted-Python
mailing list