[Twisted-Python] HTTP Agent persistent connections not closed for some HTTPS sites

[Adi Roiban]

Hi,

It looks like when connected to some  HTTPS servers, the TLS
connection is not successfully closed.

With sites like google.com, the connection is successfully closed.

But with Office365 sharepoint.com sites the TLS shutdown is not completed.

Has anyone else observed this behaviour ?

I have observed this while running some end to end tests in which the
pool.closeCachedConnections() deferred was not called, even after a
generous amount of seconds :)

--------

the TLSMemoryBIOProtocol._tlsConnection.shutdown() will return False
and the connection is left open.

Future calls to tlsConnection will result in WantReadError

>From my understanding WantReadError means that more data needs to be
read and passed to the protocol.

And If I try to read from the raw socket I get this

data = TLSMemoryBIOProtocol.transport.socket.recv(1000)
*** error: [Errno 11] Resource temporarily unavailable

and the socket is never made available  and in the end it is closed with

data = TLSMemoryBIOProtocol.transport.socket.recv(100)
*** error: [Errno 104] Connection reset by peer

-----------

Here is a gits with the example code and some example output
https://gist.github.com/adiroiban/346dd455094d1762f0e69e9812309ad6

The gist also contains a patched TLSMemoryBIOProtocol._shutdownTLS
which will close the connection after some time.

PS: TLSMemoryBIOProtocol._tlsConnection.shutdown() make reference to
this issue https://github.com/pyca/pyopenssl/issues/91... but the
issue is now closed

Thanks!
-- 
Adi Roiban