[Twisted-Python] Freezing Ampoule or the Invasion of Infinite Zombie Processes

[Burak Nehbit]

spawnProcess is somewhat low-level.  It would be very nice indeed if Twisted had some sort of Service object you could easily use for spawning worker processes, along the lines of Ampoule, that would take care of all the platform-specific nonsense of setting up communication with subprocesses, spawning them, monitoring them, figuring out the right Python to spawn, the right environment variables to get sys.path correct, and other things, such as...


Ampoule is very close. Any chance of polishing and absorbing it into Twisted? Parallel computing is the weakest point of entire Python ecosystem, and Ampoule is so very close to the perfection, unless, like me, you end up getting stuck on a corner case. 



For this, I am almost entirely sure that the imports are failing, most probably because it’s trying to import the system python, not the one PyInstaller packages up. I have no idea how to fix it though. I’m feeling rather guilty about it, Aether v1.1 is a vastly improved version, but  I can’t package it, so the release is off for a while. For personal reasons (Moved to SF, started a new full–time job) I am unlikely to find enough time to debug such a complex issue in the near future, while the code is fading from my memory and rotting in drive. Such a shame.

 

Best,

Burak






From: Glyph Lefkowitz glyph at twistedmatrix.com
Reply: Twisted general discussion twisted-python at twistedmatrix.com
Date: March 10, 2014 at 11:39:36 AM
To: Twisted general discussion twisted-python at twistedmatrix.com
Subject:  Re: [Twisted-Python] Freezing Ampoule or the Invasion of Infinite	Zombie Processes  


On Mar 9, 2014, at 12:31 AM, Burak Nehbit <burak at nehbit.net> wrote:
I agree. Though it’s also very useful as an drop–in implementation of a Twisted spawnProcess for one single other process.

spawnProcess is somewhat low-level.  It would be very nice indeed if Twisted had some sort of Service object you could easily use for spawning worker processes, along the lines of Ampoule, that would take care of all the platform-specific nonsense of setting up communication with subprocesses, spawning them, monitoring them, figuring out the right Python to spawn, the right environment variables to get sys.path correct, and other things, such as...

It sounds like the basic process support included in Twisted, 
reactor.spawnProcess, could address your requirements. 
I have just found your answer here: 

https://stackoverflow.com/questions/6678763/simple-example-with-spawnprocess

which seems useful. If nothing else works, I will try to reduce complexity by converting this to bare spawnProcess. 

Any ideas on what might be going wrong in my current stack? After all, processes being repeatedly created is just a symptom of something going wrong in the process very early, so ampoule tries to spawn another one. I am mainly looking to find a way to see the exception and where it happens. Twisted’s logging module seem to kick in too late. 

Twisted's logging typically gets initialized after all modules are imported.  A tool like Ampoule really ought to set up logging as early as possible in its child, and handle the case where it's not getting set up on the parent side.

The important thing to do is to make sure you have an errReceived callback that actually prints or logs the stderr it receives; there's likely a traceback in there which you may have missed if the process is just crashing.

-glyph
- smime.p7s, 6 KB
_______________________________________________  
Twisted-Python mailing list  
Twisted-Python at twistedmatrix.com  
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20140311/10955440/attachment-0002.html>