[Twisted-Python] inlineCallbacks vs explicit callbacks: performance concerns?

[Paul Goins]

Question: Is there a performance related reason why Twisted core does
not use inlineCallbacks?

-----

I'm trying to figure out how to reduce the CPU load of an app.  I've
already done a lot of profiling and have trimmed a lot of fat, but we're
still not getting as much throughput as we'd like.  We don't seem to
really be I/O bound in our application, but mostly CPU bound.

One thing we use a lot in our code is the inlineCallbacks decorator.
It's also used by txAMQP, at least in the 0.2 version we are currently
using.  However, I notice it doesn't seem to really be used anywhere in
the Twisted core code; I only see its definition and the defgen test
which uses it.

I'm curious of the reason why Twisted core does not use it.  Is it
performance related?  Compatibility with older Python versions?  Or
something else?

Thanks.

- Paul Goins

----

For those curious, here's what I'm currently facing:

If I count total "internal time" per module (NOT cumulative time), the
current hotspot is twisted.internet.defer with 27.88% of all CPU time.
The per-function breakdown is something like this:

======================= =====================
Line(Function)          Time (total time pct)
======================= =====================
314(_runCallbacks)      7.657
733(_inlineCallbacks)   6.246
833(unwindGenerator)    2.864
172(addCallbacks)       2.103
291(_startRunCallbacks) 1.940
206(addBoth)            1.369
766(gotResult)          1.165
233(callback)           1.014
======================= =====================

Twisted version: 8.2.0.  Profiler: cProfile.

I'm trying to find the best way of getting this down.  It seems like
reducing the raw number of callbacks is probably the best way if
possible.  However, unwindGenerator/_inlineCallbacks by itself takes
almost 10% of the time... hence the above question.