Overview
This document explains how you can use Twisted to implement network protocol parsing and handling for TCP servers (the same code can be reused for SSL and Unix socket servers). There is a separate document covering UDP.
Your protocol handling class will usually subclass twisted.internet.protocol.Protocol
. Most
protocol handlers inherit either from this class or from one of
its convenience children. An instance of the protocol class
is instantiated per-connection, on demand, and will go
away when the connection is finished. This means that
persistent configuration is not saved in the
Protocol
.
The persistent configuration is kept in a Factory
class, which usually inherits
from twisted.internet.protocol.Factory
. The buildProtocol
method of the Factory
is used to create
a Protocol
for each new connection.
It is usually useful to be able to offer the same service on
multiple ports or network addresses. This is why
the Factory
does not listen to connections, and in
fact does not know anything about the
network. See the endpoints
documentation for more information,
or twisted.internet.interfaces.IReactorTCP.listenTCP
,
and the other IReactor*.listen*
APIs for the lower
level APIs that endpoints are based on.
This document will explain each step of the way.
Protocols
As mentioned above, this, along with auxiliary classes and functions, is where most of the code is. A Twisted protocol handles data in an asynchronous manner: the protocol responds to events as they arrive from the network; the events arrive as calls to methods on the protocol.
Here is a simple example:
1 2 3 4 5 6
from twisted.internet.protocol import Protocol class Echo(Protocol): def dataReceived(self, data): self.transport.write(data)
This is one of the simplest protocols. It simply writes back whatever is written to it, and does not respond to all events. Here is an example of a Protocol responding to another event:
1 2 3 4 5 6 7
from twisted.internet.protocol import Protocol class QOTD(Protocol): def connectionMade(self): self.transport.write("An apple a day keeps the doctor away\r\n") self.transport.loseConnection()
This protocol responds to the initial connection with a well known quote, and then terminates the connection.
The connectionMade event is usually where set up of the
connection object happens, as well as any initial greetings (as
in the QOTD protocol above, which is actually based on RFC
865). The connectionLost
event is where tearing down of any
connection-specific objects is done. Here is an example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
from twisted.internet.protocol import Protocol class Echo(Protocol): def __init__(self, factory): self.factory = factory def connectionMade(self): self.factory.numProtocols = self.factory.numProtocols+1 self.transport.write( "Welcome! There are currently %d open connections.\n" % (self.factory.numProtocols,)) def connectionLost(self, reason): self.factory.numProtocols = self.factory.numProtocols-1 def dataReceived(self, data): self.transport.write(data)
Here connectionMade
and
connectionLost
cooperate to keep a count of the
active protocols in a shared object, the factory. The factory must
be passed to Echo.__init__
when creating a new
instance. The factory is used to share state that exists beyond the
lifetime of any given connection. You will see why this object is
called a "factory" in the next section.
loseConnection() and abortConnection()
In the code above, loseConnection
is called immediately
after writing to the transport. The loseConnection
call will
close the connection only when all the data has been written by Twisted
out to the operating system, so it is safe to use in this case without
worrying about transport writes being lost. If
a producer is being used with the
transport, loseConnection
will only close the connection once
the producer is unregistered.
In some cases, waiting until all the data is written out is not what we
want. Due to network failures, or bugs or maliciousness in the other side
of the connection, data written to the transport may not be deliverable,
and so even though loseConnection
was called the connection
will not be lost. In these cases, abortConnection
can be
used: it closes the connection immediately, regardless of buffered data
that is still unwritten in the transport, or producers that are still
registered. Note that abortConnection
is only available in
Twisted 11.1 and newer.
Using the Protocol
In this section, you will learn how to run a server which uses your
Protocol
.
Here is code that will run the QOTD server discussed earlier:
1 2 3 4 5 6 7 8 9 10 11 12
from twisted.internet.protocol import Factory from twisted.internet.endpoints import TCP4ServerEndpoint from twisted.internet import reactor class QOTDFactory(Factory): def buildProtocol(self, addr): return QOTD() # 8007 is the port you want to run under. Choose something >1024 endpoint = TCP4ServerEndpoint(reactor, 8007) endpoint.listen(QOTDFactory()) reactor.run()
In this example, I create a protocol Factory
. I want to tell this
factory that its job is to build QOTD protocol instances, so I set its
buildProtocol
method to return instances of the QOTD class. Then, I want to listen
on a TCP port, so I make a TCP4ServerEndpoint
to identify the
port that I want to bind to, and then pass the factory I just created to
its listen
method.
Because this is a short example, nothing else has yet started up the
Twisted reactor. endpoint.listen
tells the reactor to handle
connections to the endpoint's address using a particular protocol, but the
reactor needs to be running in order for it to do anything.
reactor.run()
starts the reactor and then waits forever for
connections to arrive on the port you've specified.
You can stop the reactor by hitting Control-C in a terminal or calling
reactor.stop
.
For more information on different ways you can listen for incoming connections, see the documentation for the endpoints API.
Helper Protocols
Many protocols build upon similar lower-level abstraction. The most popular in internet protocols is being line-based. Lines are usually terminated with a CR-LF combinations.
However, quite a few protocols are mixed - they have line-based sections and then raw data sections. Examples include HTTP/1.1 and the Freenet protocol.
For those cases, there is the LineReceiver
protocol. This protocol dispatches to two different event
handlers - lineReceived
and
rawDataReceived
. By default, only
lineReceived
will be called, once for each line.
However, if setRawMode
is called, the protocol
will call rawDataReceived
until
setLineMode
is called, which returns it to using
lineReceived
. It also provides a method,
sendLine
, that writes data to the transport along
with the delimiter the class uses to split lines (by default,
\r\n
).
Here is an example for a simple use of the line receiver:
1 2 3 4 5 6 7 8 9 10 11
from twisted.protocols.basic import LineReceiver class Answer(LineReceiver): answers = {'How are you?': 'Fine', None : "I don't know what you mean"} def lineReceived(self, line): if self.answers.has_key(line): self.sendLine(self.answers[line]) else: self.sendLine(self.answers[None])
Note that the delimiter is not part of the line.
Several other, less popular, helpers exist, such as a netstring based protocol and a prefixed-message-length protocol.
State Machines
Many Twisted protocol handlers need to write a state machine to record the state they are at. Here are some pieces of advice which help to write state machines:
- Don't write big state machines. Prefer to write a state machine which deals with one level of abstraction at a time.
- Don't mix application-specific code with Protocol handling code. When the protocol handler has to make an application-specific call, keep it as a method call.
Factories
Simpler Protocol Creation
For a factory which simply instantiates instances of a
specific protocol class, there is a simpler way to implement the factory.
The default implementation of the buildProtocol
method calls
the protocol
attribute of the factory to create
a Protocol
instance, and then sets an attribute on it
called factory
which points to the factory
itself. This lets every Protocol
access, and possibly
modify, the persistent configuration. Here is an example that uses these
features instead of overriding buildProtocol
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
from twisted.internet.protocol import Factory, Protocol from twisted.internet.endpoints import TCP4ServerEndpoint from twisted.internet import reactor class QOTD(Protocol): def connectionMade(self): # self.factory was set by the factory's default buildProtocol: self.transport.write(self.factory.quote + '\r\n') self.transport.loseConnection() class QOTDFactory(Factory): # This will be used by the default buildProtocol to create new protocols: protocol = QOTD def __init__(self, quote=None): self.quote = quote or 'An apple a day keeps the doctor away' endpoint = TCP4ServerEndpoint(reactor, 8007) endpoint.listen(QOTDFactory("configurable quote")) reactor.run()
Factory Startup and Shutdown
A Factory has two methods to perform application-specific
building up and tearing down (since a Factory is frequently
persisted, it is often not appropriate to do them in __init__
or __del__
, and would frequently be too early or too late).
Here is an example of a factory which allows its Protocols to write to a special log-file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class LoggingProtocol(LineReceiver): def lineReceived(self, line): self.factory.fp.write(line+'\n') class LogfileFactory(Factory): protocol = LoggingProtocol def __init__(self, fileName): self.file = fileName def startFactory(self): self.fp = open(self.file, 'a') def stopFactory(self): self.fp.close()
Putting it All Together
As a final example, here's a simple chat server that allows users to choose a username and then communicate with other users. It demonstrates the use of shared state in the factory, a state machine for each individual protocol, and communication between different protocols.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver from twisted.internet import reactor class Chat(LineReceiver): def __init__(self, users): self.users = users self.name = None self.state = "GETNAME" def connectionMade(self): self.sendLine("What's your name?") def connectionLost(self, reason): if self.users.has_key(self.name): del self.users[self.name] def lineReceived(self, line): if self.state == "GETNAME": self.handle_GETNAME(line) else: self.handle_CHAT(line) def handle_GETNAME(self, name): if self.users.has_key(name): self.sendLine("Name taken, please choose another.") return self.sendLine("Welcome, %s!" % (name,)) self.name = name self.users[name] = self self.state = "CHAT" def handle_CHAT(self, message): message = "<%s> %s" % (self.name, message) for name, protocol in self.users.iteritems(): if protocol != self: protocol.sendLine(message) class ChatFactory(Factory): def __init__(self): self.users = {} # maps user names to Chat instances def buildProtocol(self, addr): return Chat(self.users) reactor.listenTCP(8123, ChatFactory()) reactor.run()
The only API you might not be familiar with
is listenTCP
. listenTCP
is
the method which connects a Factory
to the network.
This is the lower-level API
that endpoints wraps for you.
Here's a sample transcript of a chat session (this is text entered by the user):
$ telnet 127.0.0.1 8123 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. What's your name? test Name taken, please choose another. bob Welcome, bob! hello <alice> hi bob twisted makes writing servers so easy! <alice> I couldn't agree more <carrol> yeah, it's great