Twisted Documentation: Extremely Low-Level Socket Operations

Introduction

Beyond supporting streams of data (SOCK_STREAM) or datagrams (SOCK_DGRAM), POSIX sockets have additional features not accessible via send(2) and recv(2). These features include things like scatter/gather I/O, duplicating file descriptors into other processes, and accessing out-of-band data.

Twisted includes a wrapper around the two C APIs which make these things possible, sendmsg and recvmsg. This document covers their usage. It is intended for Twisted maintainers. Application developers looking for this functionality should look for the high-level APIs Twisted provides on top of these wrappers.

sendmsg

sendmsg(2) exposes nearly all sender-side functionality of a socket. For a SOCK_STREAM socket, it can send bytes that become part of the stream of data being carried over the connection. For a SOCK_DGRAM socket, it can send bytes that become datagrams sent from the socket. It can send data from multiple memory locations (gather I/O). Over AF_UNIX sockets, it can copy file descriptors into whichever process is receiving on the other side. The wrapper included in Twisted, send1msg, exposes many (but not all) of these features. This document covers the usage of the features it does expose. The alternate spelling for the wrapper is used to indicate the primary limitation, which is it that the interface supports sending only one iovec at a time.

recvmsg

Likewise, recvmsg(2) exposes nearly all the receiver-side functionality of a socket. It can receive stream data over from a SOCK_STREAM socket or datagrams from a SOCK_DGRAM socket. It can receive that data into multiple memory locations (scatter I/O), and it can receive those copied file descriptors. The wrapper included in Twisted, recv1msg, exposes many (but not all) of these features. This document covers the usage of the features it does expose. The alternate spelling for the wrapper is used to indicate the primary limitation, which is that the interface supports receiving only one iovec at a time.

Sending And Receiving Regular Data

sendmsg can be used in a way which makes it equivalent to using the send call. The first argument to sendmsg is (in this case and all others) a file descriptor over which to send the data. The second argument is a string giving the data to send.

On the other end, recvmsg can be used to replace a recv call. The first argument to recvmsg is (again, in all cases) a file descriptor over which to receive the data. The second argument is an integer giving the maximum number of data to receive.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

"""
Demonstration of sending bytes over a TCP connection using sendmsg.
"""

from socket import socketpair

from twisted.python.sendmsg import send1msg, recv1msg

def main():
    foo, bar = socketpair()
    sent = send1msg(foo.fileno(), "Hello, world")
    print "Sent", sent, "bytes"
    (received, flags, ancillary) = recv1msg(bar.fileno(), 1024)
    print "Received", repr(received)
    print "Extra stuff, boring in this case", flags, ancillary

if __name__ == '__main__':
    main()

- listings/sendmsg/send_replacement.py

Copying File Descriptors

Used with an AF_UNIX socket, sendmsg send a copy of a file descriptor into whatever process is receiving on the other end of the socket. This is done using the ancillary data argument. Ancillary data consists of a list of three-tuples. A three-tuple constructed with SOL_SOCKET, SCM_RIGHTS, and a platform-endian packed file descriptor number will copy that file descriptor.

File descriptors copied this way must be received using a recvmsg call. No special arguments are required to receive these descriptors. They will appear, encoded as a native-order string, in the ancillary data list returned by recvmsg.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

"""
Demonstration of copying a file descriptor over an AF_UNIX connection using
sendmsg.
"""

from os import pipe, read, write
from socket import SOL_SOCKET, socketpair
from struct import unpack, pack

from twisted.python.sendmsg import SCM_RIGHTS, send1msg, recv1msg

def main():
    foo, bar = socketpair()
    reader, writer = pipe()

    # Send a copy of the descriptor.  Notice that there must be at least one
    # byte of normal data passed in.
    sent = send1msg(
        foo.fileno(), "\x00", 0,
        [(SOL_SOCKET, SCM_RIGHTS, pack("i", reader))])

    # Receive the copy, including that one byte of normal data.
    data, flags, ancillary = recv1msg(bar.fileno(), 1024)
    duplicate = unpack("i", ancillary[0][2])[0]

    # Demonstrate that the copy works just like the original
    write(writer, "Hello, world")
    print "Read from original (%d): %r" % (reader, read(reader, 6))
    print "Read from duplicate (%d): %r" % (duplicate, read(duplicate, 6))

if __name__ == '__main__':
    main()

- listings/sendmsg/copy_descriptor.py