[Twisted-Python] Some things I've learned: safer callbacks, better t.p.context

[Glyph Lefkowitz]

> On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobconway at gmail.com> wrote:
> 
> > This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow
> 
> Well, it's not something we've announced yet through any official channel, but we had to solve the context propagation problem at Atlassian to instrument our services with traceable logging. We open sourced our solution at https://bitbucket.org/hipchat/txlocal <https://bitbucket.org/hipchat/txlocal>. The answer for us was an extension for the reactor, thread pool, and inline callbacks that maintain the needed state. There's a readme with some insight into how we tool our services.
> 
This is (A) very cool, and (B) making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :).  I'm super impressed that you tracked the introduction of twisted._threads and support both old- and new-style thread pools!
> We've had it on our backlog to address the mailing list and, possibly even, discuss what it would take to put this into Twisted. I guess now is as good if a time as any. Feel free to spin off a another thread or reach out to me off list with any questions or feedback
> 
This seems like as good a time to talk about it as any!  Integrating this into the core in some fashion would be good, but I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that.

Speaking of performance - I found the long digression on "Don't Switch In The Core" interesting, since we actually _do_ tracking this kind of context already, for logging.  I was a little surprised you didn't integrate with this at all.

To be fair, this is something that our friends over at PyPy have been bugging us about since forever; when you're benchmarking raw wire speed it does tend to show up in profiling.

I also take it from the performance notes that you're not using PyPy?  __slots__ shouldn't make much of a difference there.  (In fact I'm given to believe it's a slight _decrease_ in performance on pypy...)

-glyph

> On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
>> On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar at itamarst.org <mailto:itamar at itamarst.org>> wrote:
>> 
>> Not been doing much Twisted lately, but have been doing async stuff
>> elsewhere, and I've learned some useful things.
> 
> Thanks for writing these up, Itamar!  This sort of reflection is rare and it's always helpful :).
> 
>> 1. Callbacks should be sync or async, but never
>> sometimes-one-sometimes-the-other. For details go read
>> http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/ <http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/>.
>> For example, Deferred.addCallback(f) really should never run f()
>> immediately.
> 
> This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
> 
> I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs).  However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop.  In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with.
> 
> I think Deferred as it is today is a pretty good compromise between the two positions.  On the one hand it is decoupled from the event loop.  On the other - and this is important - no Deferred-returning API will ever call your callbacks synchronously.  Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:
> 
> self.manipulateSomeStateForSetup()
> d = doSomethingPotentiallySynchronous()
> self.manipulateSomeStateForProcessing()
> d.addCallback(completeOperation)
> 
> As a caller, you can always decide whether you can safely be re-entered or not.  In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine.  In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
> 
>> 3.
> 
> What happened to '2'? :)
> 
>> By instrumenting all callbacks it manages, which may or may not
>> require item #1, Twisted can have a context that automatically follows
>> callbacks. Node has this and it is extremely useful.
>> http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/ <http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/>
>> is best summary I've found with a bit of searching.
> 
> This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out.  This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow.  I will watch this talk intently; if #1 really is required to address this, my opinion might change.  A PR would be intensely appreciated.
> 
> -glyph
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com <mailto:Twisted-Python at twistedmatrix.com>
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20161018/abe895f6/attachment-0002.html>