Other wx/twisted methods (was Re: [Twisted-Python] removing unsupported reactors in twisted 2.6: qt, corefoundation, threadedselect, wx)

[Stephen Waterbury]

glyph at divmod.com wrote:
> On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury 
> <waterbug at pangalactic.us> wrote:
> 
>> If I can get that to work (which I haven't had time to play with yet),
>> it would seem to offer several advantages:  (1) running wx and twisted
>> event loops in completely separate processes, (2) making relatively
>> minimal use of threads in the wx process (well-supported by the wx
>> libraries) to drive Pyro, and using the usual twisted paradigm in
>> the twisted process.  I would appreciate any advice, but I don't see
>> any show-stoppers.  I would do the Pyro communications on 127.0.0.1,
>> since Pyro has essentially zero security, keeping it completely within
>> the local machine.  Twisted would handle all external protocols.
> 
> Thanks for the suggestion, Stephen.  I don't think I like this approach 
> though.
> 
> The major reason is that it introduces a process boundary between 
> Twisted code and WX code.

I see that as a *good* thing.  ;)

> In every other toolkit Twisted supports,

... of which there's really only *one* (GTK) that's at all
interesting to anyone doing cross-platform apps ... and
even that one, IMO, isn't very cross-platform, although
that could certainly be argued either way ...

> you 
> can handle GUI events and network events in the same process, in the 
> same thread.   That makes it possible to have individual state machines 
> which can emit output in the form of either GUI updates and writes to 
> streams, and you can similarly receive GUI events or data from the 
> network as input.

Okay.  I'm willing to do without that.

> Keep in mind that each of the toolkits that WX backends to already *has* 
> either a supported reactor or a proof of concept for Twisted 
> integration: GTK+, the Win32 event loop, and Cococa.  It's not that we 
> can't get network I/O notifications from the platform.  The real problem 
> here is that WX obscures the underlying APIs for integrating sockets.  
> (I say "obscures" rather than "does not wrap" because the last time I 
> tried to use gtk reactor with a gtk build of wx, the process segfaulted 
> after emitting a pile of dynamic linker errors.  I have no idea what 
> would happen on Windows or the Mac.)  Whatever we're going to do at this 
> point without getting some new code from the WX developers is going to 
> be a workaround.

I think *that* is the real problem ... not technical, just interest
from the right people.  I've waited enough years (about 5) that
I've given up on that.

> That said, the mechanism that you propose may be the best way to 
> practically integrate Twisted and WX today.  If it is, I think that it 
> would work better as a separate project, because it is a workaround with 
> a lot of pointy edges and not a real solution.

What's reality?  :)  OTOH ...
I completely agree that it should not be part of Twisted
at all ... but if I succeed with it, I'll certainly document what I
do.

> One major reason I'd like a WX reactor to continue to exist in some form 
> is OSAF's usage of Twisted and WX.  They have a similarly baroque model 
> for handing data back and forth between Twisted code and GUI code, ...

... and everything else (sorry, couldn't resist ;) ...

> ... and 
> I'd love to have something to propose to them that was (A) supported by 
> Twisted (B) well tested, and (C) didn't involve lots of potential race 
> conditions, as all shared-state-threading code was likely to do.  I 
> haven't looked at it in a while, but if I remember correctly their 
> separation between Twisted code and GUI code was fairly clean and 
> well-thought-out,

... hmmm ... I've been on all the OSAF Chandler/Scooby/Cosmo lists
since April (and that's a lot of freakin' mail ;), and not *once* in
that interval has "twisted" occurred in the subject line of a
message, and no substantive discussion of it in the bodies either.

I know about their zanshin twisted-based framework for WebDAV/CalDAV,
and that it's used in their Sharing architecture, which is used by
the Chandler wx GUI, but I just updated my chandler svn checkout, and
I counted 17 occurrences of the zanshin "blockUntil" function in the
Sharing.py module ... if it's so clean, how come they need those?
(In zanshin's docs, "blockUntil" is billed as "for test purposes" ... ;)

> but the separation would have been unnecessary if wx 
> had native Twisted integration.

Well, yeah, but I don't think OSAF has the resources to put into it
either, and Robin *works* for OSAF.

> There are other problems with a Pyro-driven Twisted subprocess, which 
> are relevant as the goal here is a well-supported and well-tested path 
> for integration.
> 
> Multiprocess communication and process spawning, especially on Windows, 
> is fraught with peril.  We've been working on reliable event-driven 
> process spawning in Twisted for years and years, and only recently have 
> we gotten to the point where the published API works reliably 
> cross-platform.  Given that Twisted would be left out of this loop, I 
> imagine the process control would be rather ad-hoc, so getting the 
> Twisted and WX processes synchronized would be problematic, and 
> extremely hard to unit test.  Although it would be slightly easier if 
> the Twisted proces is 'on top', you still have to inject test code into 
> both the parent and child processes, and have a way to verify the state 
> of both.
> 
> The last release of pyro is almost a year old.  That suggests it is not 
> actively maintained.

Or else it just works.  ;)

> Pyro doesn't have a buildbot, or (as far as I can tell from the release) 
> any automated tests of its own.  That would make it a wash in terms of 
> tested, robust integration support.

As if there's going to be some other "tested, robust integration
support" ... nothing on the horizon that I can see (and as I say,
I've been watching for a long time).

> You'd have to re-implement the Pyro protocol on top of Twisted to 
> communicate with it without spawning lots of threads.  That's a bunch of 
> new code which is going to need a home, again, possibly a whole new 
> project.  Even after that's done, Pyro is going to be spawning threads 
> and blocking inside the WX process because the Twisted implementation 
> isn't running there.

Process spawning!  Rusty code!  Spawning threads, *and* blocking!
My, you do paint a rosy picture.  All your handwaving and FUD makes
me want to try it, because it's probably quicker and easier than
discussing it.  But thanks for the warnings, anyway!  ;)

Cheers,
Steve