[Twisted-Python] suggestion for tracking down cause of 2 bugs sought: possibly threads / defer.DeferredSemaphore ?

[Alec Matusis]

> > Is there any way to profile memory usage by twisted
> in a
> > granular fashion ?  I'd like to figure out where this memory usage
> > is ( it grows steadily to ~1.7gb after 8 hours ).

We found twisted manhole very useful for this. Inspect all your objects via
manhole on the production server.
Also, in one instance, we had to force python garbage collector to collect
every 10 min: for some reason: without this, the RSS size just grew and
grew...

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Jonathan Vanasco
> Sent: Wednesday, March 21, 2007 12:34 PM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] suggestion for tracking down cause of 2
> bugs sought: possibly threads / defer.DeferredSemaphore ?
> 
> 
> On Mar 21, 2007, at 11:32 AM, Jonathan Vanasco wrote:
> 
> 
> > 	2)
> > 		I'm experiencing an obscene amount of process growth.  I'm
> fairly
> > certain that its from constant errors in urllib2 , which has been
> > nothing of a nightmare with countless known bugs and patches
> > ( downloading takes place via Mechanize , which is wrapped via
> > urllib2 ), and makes me consider just wrapping wget or curl.   but
> > this could very well be threading related.
> >
> > 		Question:
> > 			Is there any way to profile memory usage by twisted
> in a
> > granular fashion ?  I'd like to figure out where this memory usage
> > is ( it grows steadily to ~1.7gb after 8 hours ).
> >
> 
> I just came across this:
> 	From: 	  mwl at point45.com
> 	Subject: 	Re: [Twisted-Python] questions about large db
queries
> and
> memory leak
> 	Date: 	February 8, 2007 8:28:59 AM EST
> 	...
> 	If you're not running Python >= 2.5.0 this might be relevant:
> 	http://evanjones.ca/python-memory.html
> 
> 
> i was on 2.4, upping to 2.5 solved that issue.  incremental growth
> of .2mb / thread instance ( which ate 1.7gb of ram in an hour or
> so ) , has been replaced by a steady 18-25mb daemon
> 
> my machine is responsive again.
> 
> // Jonathan Vanasco
> 
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - - - - - - - - -
> | SyndiClick.com
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - - - - - - - - -
> |      FindMeOn.com - The cure for Multiple Web Personality Disorder
> |      Web Identity Management and 3D Social Networking
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - - - - - - - - -
> |      RoadSound.com - Tools For Bands, Stuff For Fans
> |      Collaborative Online Management And Syndication Tools
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - - - - - - - - -
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python