[Twisted-Python] HTTP Agent persistent connections not closed for some HTTPS sites

[Adi Roiban]

On 6 March 2017 at 08:17, Cory Benfield <cory at lukasa.co.uk> wrote:
>
> On 5 Mar 2017, at 14:25, Tristan Seligmann <mithrandi at mithrandi.net> wrote:
>
> On Sun, 5 Mar 2017 at 15:36 Adi Roiban <adi at roiban.ro> wrote:
>>
>> I have observed this while running some end to end tests in which the
>> pool.closeCachedConnections() deferred was not called, even after a
>> generous amount of seconds :)
>
>
> The code to abort an HTTP client connection is here:
>
> https://github.com/twisted/twisted/blob/twisted-17.1.0/src/twisted/web/_newclient.py#L1657
>
> This calls loseConnection which for a TLS connection will try to do a TLS
> shutdown under most circumstances (in some cases it can't, and will
> abortConnection on the underlying transport instead). If the remote end has
> stopped responding to the connection, I think this may end up hanging
> forever.
>
> I think this code should either call abortConnection directly, or set a
> timer which will abort the connection after a little while if a clean
> shutdown from loseConnection has not completed yet. I’
>
>
> Yeah, this looks right. We hit it in the server side code too: it’s a very
> unintuitive API in that sense.

I am not sure about which code are we talking here.
The specific HTTP11ClientProtocol which will fix only the HTTP client
part or the generic TLSMemoryBIOProtocol code which might fix any TLS
connection?

I have observed this issue on the server side for the case in which a
TLS connection is opened and then closed  without transferring any
data ... and then the handshake will not have the chance to complete.

I think that this is the same issue which was reported for Scrapy

-- 
Adi Roiban