[Twisted-Python] My memory leak, diagnosed

[glyph at divmod.com]

On 08:58 am, matt at madebykite.com wrote:
>I just wanted to put your minds at rest regarding what I though might 
>have been a problem with the epoll reactor. As always, inferring 
>anything from a production machine is not desirable and it seems that 
>my assumptions about epoll were based on coincidence.

Thanks for the thorough diagnosis and detailed ticket!  And I'm glad to 
hear the issue is not with epoll, I'd like our users to be able to rest 
easy when upgrading to a "good" reactor :).  While twisted.web.proxy 
should really work well, I'm a bit less concerned to hear that it might 
have some issues.  Especially if you're planning on fixing them.

Looking at the ticket, though, it appears that web.proxy might be okay 
too, and the memory growth is just the process reaching its natural 
equilibrium, or the source is somewhere else?
>Unfortunately, my workaround involved rewriting my proxy code without 
>using twistd.web. I'll happily contribute a patch back to the 
>t.w.proxy code in time but for the moment it doesn't support 
>persistent connections (I wanted to keep it simple). I also feel that 
>t.w.proxy would be better served by making use of a reciprocal 
>IProducer / IConsumer arrangement between the ProxyServer protocol  and 
>the ProxyClient protocol which is something that my workaround  does 
>not implement.

Not implementing IProducer/IConsumer will, ironically, lead to more 
memory consumption in the long term, since the producer will overload 
the consumer's buffer if the consumer's not fast enough.