[Twisted-Python] Woven application, objects not collected.

[Andrew Bennetts]

On Tue, Nov 25, 2003 at 01:10:09PM +0100, Syver Enstad wrote:
> 
> I have a serious problem with some woven applications where deferreds
> and their results returned from wmfactory_ methods are not garbage
> collected. They just stack up in gc.garbage. The python documentation
> states that the contents of gc.garbage are objects that are part of a
> cycle can't be collected because they have a __del__ method.
> 
> Any help on this issue would be really appreciated.

Deferreds have a __del__ method, used to log unhandled errors, which has
often proved to be a valuable debugging aid (people often forget to
.addErrback(log.err)).

Unfortunately, this means that Deferreds in reference cycles can't be
automatically garbage collected.  Can you boil this down to a minimal test
case?  This sounds like the probable cause of a memory leak some users have
reported with Twisted Web.

The trick will be to figure out what the reference cycle is, and breaking
it.  It may help to do:
    twisted.internet.defer.Deferred.debug = True
and then inspect the .creator attribute of the Deferreds the end up in
gc.garbage.  You could also try manually poking around with gc.get_referrers
on those Deferreds.

If we can find a good enough test case, we might be able to write a unit
test to trigger it and then check gc.garbage, so that we can include it in
the test suite and stop it from happening again.

I'm very keen to find out what's causing this -- let me know if there's
anything I can help with.

-Andrew.