[Twisted-Python] Twisted WebSockets Branch

[Tobias Oberstein]

Hi Rob,

if your main concern is robust WebSocket on Twisted, why not use Autobahn (https://pypi.python.org/pypi/autobahn) ?

Autobahn supports wrapping arbitrary stream-based protocols on top of WebSocket, with full support for advanced features like WebSocket compression and auto-fragmentation plus Twisted endpoints:

https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/websocket/wrapping

Note: Wrapping arbitrary stream-based protocols in WebSocket requires proper WebSocket subprotocol negotiation and handling. Autobahn follows the scheme established by https://github.com/kanaka/websockify for which you find e.g. JS libraries running in a browser.

Autobahn also supports running WebSocket as a resource in a Twisted Web resource tree: https://github.com/tavendo/AutobahnPython/blob/master/examples/twisted/websocket/echo_site/server.py

Disclosure: I wrote this stuff. So I am biased;) Glyph at one point mentioned that he likes to see "basic WebSocket support" built into Twisted. Bringing the branch to the full level of Autobahn rgd. features, compliance, robustness etc will be a lot of work though. Merging Autobahn into Twisted is unlikely to happen / doesn't make sense for different reasons I guess.

/Tobias

Von: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] Im Auftrag von Rob Oakes
Gesendet: Freitag, 22. August 2014 04:24
An: twisted-python at twistedmatrix.com
Cc: roakes at gurulabs.com
Betreff: [Twisted-Python] Twisted WebSockets Branch

Dear Twisted Developers,

As a quick introduction, my name is Rob Oakes. I'm the lead developer for a company called Guru Labs. I'm writing for two reasons:

# Reason 1: Thank You

First and foremost, I'm writing to express thanks for creating an excellent framework. We use Twisted extensively in our infrastructure, and it is typically our go-to tool for any sort of custom server.

The other two reasons have to do with some questions about the development of Twisted. Before I dive into those, however, let me provide some background.

For the better part of a year, we've been using some code out of the websockets (twisted.web.websockets.WebsocketsResource) branch to wrap a trio of custom protocols we use in one of our web based products. I know that the code is still pre-release, but we've generally found it to be stable and work very well.

# Reason Two: Websockets Development

This brings me to the second reason I'm writing. Over the past year, it seems as though the development on the Twisted websockets branch has stalled. We would like to unstall it. For this reason, Guru Labs would be interested in:

1.) Contracting with one of the core Twisted devs of Twisted who might be interested in finishing the implementation of the websockets wrappers (resolving the issues described in ticket 4173) or another party. We are happy to pay hourly rates, set a bounty, make a donation to the Twisted project ... whatever.

A stable implementation of websockets available in Twisted Web is a priority to us, and if it's a matter of money, we are happy to throw money where it might be needed. (If this of interest, please contact me off-list at roakes at gurulabs.com<mailto:roakes at gurulabs.com>.)

or

2.) Assigning a Guru Labs developer (probably me) who might complete the work.

I've been studying the issues which are still open (specifically 4173), and the various branches associated with websocket development (websocket-4173-3 and websocket-4173-4) and it seems that most of the major concerns are related to the manner in which websocket connections are closed.

Assuming that someone else doesn't step forward ... I've already merged the most recent version of trunk with these branches, but I've found myself with several questions on how to best continue with the work.

a. Which of the two websocket branches should be used as the basis for new work?

On GitHub, websocket-4173-4 is marked as closed with a note telling the contributor to see the contribution guidelines. There are also several additions to the code which do not follow Twisted conventions (as I understand them). The last set of commits also seem to come from approximately the same time.

websocket-4173-4 includes code which attempts to resolve issues noted in 4173 that is not present in 4173-3, but there aren't really any comments to determine if this should be incorporated or not. At this point, I'm really not familiar enough with the code to draw my own conclusion.

b. In general, the protocol wrapper seems to work quite well. However when merged with the most recent version of trunk (post twisted 14), I've been seeing frequent  unhandled deferred errors. The most common is:

"twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost"

It happens when:

* Connections are closed from the server, using transport.loseConnection
* Connections do not transmit a "close" frame prior to disconnecting
* To reiterate, this issue only started appearing after merging the websocket code with Twisted 14. The previous version of Twisted we were using (Twisted 12.3 worked flawlessly.)

No error is raised if the client correctly closes the connection or when using non-browser based clients (like the Python ws4py websocket client).

We mostly see the exception when the objects are garbage collected (based upon the deferred documentation at http://twistedmatrix.com/documents/14.0.0/core/howto/defer.html). Put another way, we see a whole string of errors upon stopping the reactor. Despite the exception, we don't see any errors in the browser client.

Also somewhat frustratingly, the traceback isn't terribly helpful. This is a fairly routine example:

```error
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.
```

Technically, the error is probably appropriate, as they appear when the connection is lost in a non-clean fashion. I am a little concerned, though, in that I haven't found a good way to catch or suppress the error. Moreover, it doesn't seem like the defferreds are getting garbage collected which seems like a memory leak waiting to happen.

Can someone clarify if this is:

* intended behavior, and if so, what might be a strategy I can use for managing the error in my wrapped protocols
* which part of the websocket code I should be looking at in order to try and fix the issue

Thoughts would be greatly appreciated.

Cheers,

Rob Oakes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20140824/819d8341/attachment-0002.html>