[Twisted-web] Guard, LastPage Memory Leak?

[Jean-Paul Calderone]

On Thu, 30 Jul 2009 01:34:26 +0200, Federico Tomassini <federicotom at yahoo.it> wrote:
>Jean-Paul Calderone wrote:
>
>>>Ok, this minimal server application proofs that a memory
>>>leak exists somewhere. This is already a good result.
>>
>> I messed around with this for a while.  Unfortunately, I am not able
>> to reproduce a memory leak using it.  I found some things which I find
>> suspicious in Nevow, but none of them explains a leak.
>
>Are you saying that you are not sure about memory leaks
>existence?

No, only that I am not able to reproduce it (I believe you when you say that
you have a server which leaks memory).

>The previous example allocates memory such way:
>
>  def afterRender(self, *a):
>    self.foo= ['a' * (10**6) for i in xrange(100)]
>
>When logout, memory remains allocated unless you
>create a destroyer method that deletes the foo
>attr before the exit, explicitley.
>
>Memory leaks appears also when rend.Page.data_bar()
>is used. Both cases seem to be related to WovenContext
>and docFactory (a lot of WovenCtx and IMacroFactory
>instances seems to stick in docFactory._cache).

Yes, this I am able to observe.  However, this leak limits itself to a single
instance.  After the first request, no further memory is leaked (when I try
the example you provided).

>In fact, this part of nevow is simply a Memory
>Blackhole. Inside docFactory._cache one can find
>various sticky objects.

Yes, but they're generally things which are needed over and over again for
every request.  That is the purpose of the cache. :)

>So, why "you are not able to reproduce it"?

When I run the example, it starts off by using about 10MB of memory.  When I
make one request of it, it jumps to 50MB of memory.  When I make a second (or
third, or any additional number) request of it, memory usage remains at
50MB.  So, that's what I mean.

>This is a *really heavy* problem. So, imho, nevow
>should work on it with energies.

I can understand how it would seem to be a very serious problem from your
perspective.  It probably makes your Nevow-based application unusable, or at
least require significant additional administrative resources.

However... none of *my* Nevow-based applications exhibit this behavior. Even
ignoring the fact that this makes it very difficult for me to try to fix the
problem (or even understand what the problem is - I still don't, by the way),
this means it's not going to work its way very far up my list of priorities -
sorry.  There are other ways to move things up my list of priorities, most
prominently, you could fund the debugging and development required to fix the
issue.  As my employer has recently become defunct, the logistics of this are
slightly complicated, but I'm sure we can work something out if you are
interested.

All that said, I'm still happy to give you my time in working on this,
provided an example I can use to reproduce the misbehavior myself.  Without
this, it's just too hard.

>Maybe the better way is to redesign WovenCtx machinery,
>as Jean-Paul suggested. But it's absolutely important
>to make an intervention.
>
>I can help. If we need to redesign WovenCtx and someone
>is able to guide my work (requiring to me precise APIs
>or Classes), you can rely on my contribution.

Some effort has been directed towards this goal already.  Element supersedes
Fragment.  It comes with a flattener which doesn't use the context (except
for backwards compatibility).  The next step is to provide a similar class to
supersede Page.  This hasn't been extremely high priority, since it's always
possible to structure all of your rendering logic as a number of Elements
with only the thinnest layer of support from Page.

Jean-Paul