twisted.web2.stream : API documentation

The stream module provides a simple abstraction of streaming data. While Twisted already has some provisions for handling this in its Producer/Consumer model, the rather complex interactions between producer and consumer makes it difficult to implement something like the CompoundStream object. Thus, this API.

The IStream interface is very simple. It consists of two methods: read, and close. The read method should either return some data, None if there is no data left to read, or a Deferred. Close frees up any underlying resources and causes read to return None forevermore.

IByteStream adds a bit more to the API: 1) read is required to return objects conforming to the buffer interface. 2) .length, which may either an integer number of bytes remaining, or None if unknown 3) .split(position). Split takes a position, and splits the stream in two pieces, returning the two new streams. Using the original stream after calling split is not allowed.

There are two builtin source stream classes: FileStream and MemoryStream. The first produces data from a file object, the second from a buffer in memory. Any number of these can be combined into one stream with the CompoundStream object. Then, to interface with other parts of Twisted, there are two transcievers: StreamProducer and ProducerStream. The first takes a stream and turns it into an IPushProducer, which will write to a consumer. The second is a consumer which is a stream, so that other producers can write to it.

Interface	IStream	A stream of arbitrary data.
Interface	IByteStream	A stream which is of bytes.
Interface	ISendfileableStream	No interface docstring; 1/1 methods documented
Class	SimpleStream	Superclass of simple streams with a single buffer and a offset and length into that buffer.
Function	mmapwrapper	Python's mmap call sucks and ommitted the "offset" argument for no discernable reason. Replace this with a mmap module that has offset.
Class	FileStream	No class docstring; 1/3 methods documented
Class	MemoryStream	A stream that reads data from a buffer object.
Class	CompoundStream	A stream which is composed of many other streams.
Function	readStream	Pass a stream's data to a callback.
Function	readAndDiscard	Read all the data from the given stream, and throw it out.
Function	readIntoFile	Read a stream and write it into a file.
Function	connectStream	Connect a protocol constructed from a factory to stream.
Function	fallbackSplit	Undocumented
Class	TruncaterStream	Undocumented
Class	PostTruncaterStream	Undocumented
Class	ProducerStream	Turns producers into a IByteStream. Thus, implements IConsumer and IByteStream.
Class	StreamProducer	A push producer which gets its data by reading a stream.
Class	ProcessStreamer	Runs a process hooked up to streams.
Function	generatorToStream	Converts a generator function into a stream.
Class	BufferedStream	A stream which buffers its data to provide operations like readline and readExactly.
Function	substream	Undocumented
Class	_StreamReader	Process a stream's data using callbacks for data and stream finish.
Class	_ProcessStreamerProtocol	Undocumented
Class	_StreamIterator	Undocumented
Class	_IteratorStream	Undocumented

def connectStream(inputStream, factory): (source)

Connect a protocol constructed from a factory to stream.

Returns an output stream from the protocol.

The protocol's transport will have a finish() method it should call when done writing.

def generatorToStream(fun): (source)

Converts a generator function into a stream.

The function should take an iterator as its first argument, which will be converted *from* a stream by this wrapper, and yield items which are turned *into* the results from the stream's 'read' call.

One important point: before every call to input.next(), you *MUST* do a "yield input.wait" first. Yielding this magic value takes care of ensuring that the input is not a deferred before you see it.

>>> from twisted.web2 import stream
>>> from string import maketrans
>>> alphabet = 'abcdefghijklmnopqrstuvwxyz'
>>>
>>> def encrypt(input, key):
...     code = alphabet[key:] + alphabet[:key]
...     translator = maketrans(alphabet+alphabet.upper(), code+code.upper())
...     yield input.wait
...     for s in input:
...         yield str(s).translate(translator)
...         yield input.wait
...
>>> encrypt = stream.generatorToStream(encrypt)
>>>
>>> plaintextStream = stream.MemoryStream('SampleSampleSample')
>>> encryptedStream = encrypt(plaintextStream, 13)
>>> encryptedStream.read()
'FnzcyrFnzcyrFnzcyr'
>>>
>>> plaintextStream = stream.MemoryStream('SampleSampleSample')
>>> encryptedStream = encrypt(plaintextStream, 13)
>>> evenMoreEncryptedStream = encrypt(encryptedStream, 13)
>>> evenMoreEncryptedStream.read()
'SampleSampleSample'

t.w.stream : module documentation