[Twisted-Python] Re: GUI responsiveness

[glyph at divmod.com]

On Thu, 15 Sep 2005 10:51:11 -0700, Jasper <jasper at peak.org> wrote:
>Nicola Larosa wrote:
>>>Worse, I don't see a satisfactory way to deal
>>>with the problem if it stems from some internal bit of Twisted taking
>>>too big of a time slice.

>>Two noticeable cases of that are (de)serialization in PB and in XML-RPC 
>>code.

There is a thing that SEDA does to make these kinds of things more reasonable.  Deferreds are sort of a degenerate case of this thing.  They don't name the thing, but I would call it "stage discipline", the act of splitting code up into what they call "stages" and making each stage idempotent except for the terminal, transaction-processing/notification-emitting stage.  I think it would be great if more of Twisted did this, but doing it is a big job.

Defgen is a slightly-less-degenerate form of this, however, using defgen on something performance critical like (de)serialization is likely to have a HUGE performance hit.

>>Is there going to be any facility, in newpb, for splitting up
>>(de)serialization in small chunks?

As warner said, "yes".

>Yes, PB (de)serialization seems to be where I get responsiveness burps too. 
>I can split up large processing chunks of my own (probably along the line of 
>JP's Cooperator), but I don't see how to readily do that to delays that are 
>internal to Twisted, without maintaining my own version or shunting all 
>networking into a seperate thread (which I'm loath to do).

"shunting all networking into a separate thread" will certainly not work, since all of the APIs that jelly calls are not thread-safe.  So you can rest easy in the knowledge that what you want to do is not only disgusting, it's impossible :).

However, the problem here is that certain code in Twisted is not as awesome as it possibly could be.  While I agree this is sad, there isn't really a solution, except "make it more awesome", which takes time, money, and effort, which nobody is contributing beyond an individual scale.

PB has been effectively unmaintained for almost 2 years now.  It is showing its age.  NewPB will fix several issues, but until someone wants to give Mr. Warner a staff and a budget for improving it, it's likely that it will have some ugly spots too.

In other words, you don't have to maintain your own version.  You just have to maintain the one in Twisted ;-).