Twisted Documentation: Getting Connected with Endpoints

Introduction

On a network, one can think of any given connection as a long wire, stretched between two points. Lots of stuff can happen along the length of that wire - routers, switches, network address translation, and so on, but that is usually invisible to the application passing data across it. Twisted strives to make the nature of the "wire" as transparent as possible, with highly abstract interfaces for passing and receiving data, such as ITransport and IProtocol.

However, the application can't be completely ignorant of the wire. In particular, it must do something to start the connection, and to do so, it must identify the end points of the wire. There are different names for the roles of each end point - "initiator" and "responder", "connector" and "listener", or "client" and "server" - but the common theme is that one side of the connection waits around for someone to connect to it, and the other side does the connecting.

In Twisted 10.1, several new interfaces were introduced to describe each of these roles for stream-oriented connections: IStreamServerEndpoint and IStreamClientEndpoint. The word "stream", in this case, refers to endpoints which treat a connection as a continuous stream of bytes, rather than a sequence of discrete datagrams: TCP is a "stream" protocol whereas UDP is a "datagram" protocol.

Constructing and Using Endpoints

In both Writing Servers and Writing Clients, we covered basic usage of endpoints; you construct an appropriate type of server or client endpoint, and then call listen (for servers) or connect (for clients).

In both of those tutorials, we constructed specific types of endpoints directly. However, in most programs, you will want to allow the user to specify where to listen or connect, in a way which will allow the user to request different strategies, without having to adjust your program. In order to allow this, you should use clientFromString or serverFromString.

There's Not Much To It

Each type of endpoint is just an interface with a single method that takes an argument. serverEndpoint.listen(factory) will start listening on that endpoint with your protocol factory, and clientEndpoint.connect(factory) will start a single connection attempt. Each of these APIs returns a value, though, which can be important.

However, if you are not already, you should be very familiar with Deferreds, as they are returned by both connect and listen methods, to indicate when the connection has connected or the listening port is up and running.

Servers and Stopping

IStreamServerEndpoint.listen returns a Deferred that fires with an IListeningPort. Note that this deferred may errback. The most common cause of such an error would be that another program is already using the requested port number, but the exact cause may vary depending on what type of endpoint you are listening on. If you receive such an error, it means that your application is not actually listening, and will not receive any incoming connections. It's important to somehow alert an administrator of your server, in this case, especially if you only have one listening port!

Note also that once this has succeeded, it will continue listening forever. If you need to stop listening for some reason, in response to anything other than a full server shutdown (reactor.stop and / or twistd will usually handle that case for you), make sure you keep a reference around to that listening port object so you can call IListeningPort.stopListening on it. Finally, keep in mind that stopListening itself returns a Deferred, and the port may not have fully stopped listening until that Deferred has fired.

Most server applications will not need to worry about these details. One example of a case where you would need to be concerned with all of these events would be an implementation of a protocol like non-PASV FTP, where new listening ports need to be bound for the lifetime of a particular action, then disposed of.

Clients and Cancelling

IStreamClientEndpoint.connect will connect your protocol factory to a new outgoing connection attempt. It returns a Deferred which fires with the IProtocol returned from the factory's buildProtocol method.

Connection attempts may fail, and so that Deferred may also errback. If it does so, you will have to try again; your protocol won't be constructed, and no further attempts will be made.

Connection attempts may also take a long time, and your users may become bored and wander off. If this happens, and your code decides, for whatever reason, that you've been waiting for the connection too long, you can call Deferred.cancel on the Deferred returned from connect, and the underlying machinery should give up on the connection. This should cause the Deferred to errback, usually with CancelledError; although you should consult the documentation for your particular endpoint type to see if it may do something different.

Although some endpoint types may imply a built-in timeout, the interface does not guarantee one. If you don't have any way for the application to cancel a wayward connection attempt, the attempt may just keep waiting forever. For example, a very simple 30-second timeout could be implemented like this:

1
2
attempt = myEndpoint.connect(myFactory)
reactor.callLater(30, attempt.cancel)

Maximizing the Return on your Endpoint Investment

Directly constructing an endpoint in your application is rarely the best option, because it ties your application to a particular type of transport. The strength of the endpoints API is in separating the construction of the endpoint (figuring out where to connect or listen) and its activation (actually connecting or listening).

If you are implementing a library that needs to listen for connections or make outgoing connections, when possible, you should write your code to accept client and server endpoints as parameters to functions or to your objects' constructors. That way, application code that calls your library can provide whatever endpoints are appropriate.

If you are writing an application and you need to construct endpoints yourself, you can allow users to specify arbitrary endpoints described by a string using the clientFromString and serverFromString APIs. Since these APIs just take a string, they provide flexibility: if Twisted adds support for new types of endpoints (for example, IPv6 endpoints, or WebSocket endpoints), your application will automatically be able to take advantage of them with no changes to its code.

Endpoints Aren't Always the Answer

For many use-cases, especially the common case of a twistd plugin which runs a long-running server that just binds a simple port, you might not want to use the endpoints APIs directly. Instead, you may want to construct an IService, using strports.service , which will fit neatly into the required structure of the twistd plugin API . This doesn't give your application much control - the port starts listening at startup and stops listening at shutdown - but it does provide the same flexibility in terms of what type of server endpoint your application will support.

It is, however, almost always preferable to use an endpoint rather than calling a lower-level APIs like connectTCP, listenTCP, etc, directly. By accepting an arbitrary endpoint rather than requiring a specific reactor interface, you leave your application open to lots of interesting transport-layer extensibility for the future.