HTML Templating with twisted.web.template

  1. A Very Quick Introduction To Templating In Python
  2. twisted.web.template - Why And How you Might Want to Use It
  3. Quoting
  4. Deferreds
  5. A Brief Note on Formats and DOCTYPEs
  6. A Bit of History

A Very Quick Introduction To Templating In Python

HTML templating is the process of transforming a template document (one which describes style and structure, but does not itself include any content) into some HTML output which includes information about objects in your application. There are many, many libraries for doing this in Python: to name a few, jinja2, django templates, and clearsilver. You can easily use any of these libraries in your Twisted Web application, either by running them as WSGI applications or by calling your preferred templating system's APIs to produce their output as strings, and then writing those strings to Request.write.

Before we begin explaining how to use it, I'd like to stress that you don't need to use Twisted's templating system if you prefer some other way to generate HTML. Use it if it suits your personal style or your application, but feel free to use other things. Twisted includes templating for its own use, because the twisted.web server needs to produce HTML in various places, and we didn't want to add another large dependency for that. Twisted is not in any way incompatible with other systems, so that has nothing to do with the fact that we use our own.

twisted.web.template - Why And How you Might Want to Use It

Twisted includes a templating system, twisted.web.template. This can be convenient for Twisted applications that want to produce some basic HTML for a web interface without an additional dependency.

twisted.web.template also includes support for Deferreds, so you can incrementally render the output of a page based on the results of Deferreds that your application has returned. This feature is fairly unique among templating libraries.

In twisted.web.template, templates are XHTML files which also contain a special namespace for indicating dynamic portions of the document. For example: 

<html xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
<body>
  <div t:render="header" />
  <div id="content">
    <p>Content goes here.</p>
  </div>
  <div t:render="footer" />
</body>
</html>
template example - listings/template-1.xml
The basic unit of templating is twisted.web.template.Element. An Element is given a way of loading a bit of markup like the above example, and knows how to correlate render attributes within that markup to Python methods exposed with twisted.web.template.renderer: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from twisted.web.template import Element, renderer, XMLFile

class ExampleElement(Element):
    loader = XMLFile('template-1.xml')

    @renderer
    def header(self, request, tag):
        return tag('Header.')

    @renderer
    def footer(self, request, tag):
        return tag('Footer.')
element example - listings/element_1.py
In order to combine the two, we must render the element. For this simple example, we can use the flattenString API, which will convert a single template object - such as an Element - into a Deferred which fires with a single string, the HTML output of the rendering process. 
1
2
3
4
5

from twisted.web.template import flattenString
from element_1 import ExampleElement
def renderDone(output):
    print output
flattenString(None, ExampleElement()).addCallback(renderDone)
rendering snippet - listings/render_1.py

This short program cheats a little bit; we know that there are no Deferreds in the template which require the reactor to eventually fire; therefore, we can simply add a callback which outputs the result. Also, none of the renderer functions require the request object, so it's acceptable to pass None through here. (The 'request' object here is used only to relay information about the rendering process to each renderer, so you may always use whatever object makes sense for your application. Note, however, that renderers from library code may require an IRequest.)

If you run it yourself, you can see that it produces the following output: 

<html>
<body>
  <div>Header.</div>
  <div id="content">
    <p>Content goes here.</p>
  </div>
  <div>Footer.</div>
</body>
</html>
rendering output 1 - listings/output-1.html
The third parameter to a renderer method is a Tag object which represents the XML element with the t:render attribute in the template. Calling a Tag adds children to the element in the DOM, which may be strings, more Tags, or other renderables such as Elements. For example, to make the header and footer bold: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from twisted.web.template import Element, renderer, XMLFile, tags

class ExampleElement(Element):
    loader = XMLFile('template-1.xml')

    @renderer
    def header(self, request, tag):
        return tag(tags.b('Header.'))

    @renderer
    def footer(self, request, tag):
        return tag(tags.b('Footer.'))
tag manipulation example - listings/element_2.py
Rendering this in a similar way to the first example would produce: 
<html>
<body>
  <div><b>Header.</b></div>
  <div id="content">
    <p>Content goes here.</p>
  </div>
  <div><b>Footer.</b></div>
</body>
</html>
tag manipulation output - listings/output-2.html
In addition to adding children, call syntax can be used to set attributes on a tag. For example, to change the id on the div while adding children: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from twisted.web.template import Element, renderer, XMLFile, tags

class ExampleElement(Element):
    loader = XMLFile('template-1.xml')

    @renderer
    def header(self, request, tag):
        return tag(tags.p('Header.'), id='header')

    @renderer
    def footer(self, request, tag):
        return tag(tags.p('Footer.'), id='footer')
attributes example - listings/element_3.py
And this would produce the following page: 
<html>
<body>
  <div id="header"><p>Header.</p></div>
  <div id="content">
    <p>Content goes here.</p>
  </div>
  <div id="footer"><p>Footer.</p></div>
</body>
</html>
attributes output - listings/output-3.html

Calling a tag mutates it, it and returns the tag itself, so you can pass it forward and call it multiple times if you have multiple children or attributes to add to it. twisted.web.template also exposes some convenient objects for building more complex markup structures from within renderer methods in the tags object. In the examples above, we've only used tags.p and tags.b, but there should be a tags.x for each x which is a valid HTML tag. There may be some omissions, but if you find one, please feel free to file a bug.

Template Attributes

t:attr tags allow you to set HTML attributes (like href in an <a href="...) on an enclosing element.

Slots

t:slot tags allow you to specify "slots" which you can conveniently fill with multiple pieces of data straight from your Python program. The following example demonstrates both t:attr and t:slot in action. Here we have a layout which displays a person's profile on your snazzy new Twisted-powered social networking site. We use the t:attr tag to drop in the "src" attribute on the profile picture, where the actual value of src attribute gets specified by a t:slot tag within the t:attr tag. Confused? It should make more sense when you see the code: 
<div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1"
    t:render="person_profile"
    class="profile">
<img><t:attr name="src"><t:slot name="profile_image_url" /></t:attr></img> 
<p><t:slot name="person_name" /></p>
</div>
slots and attributes template - listings/slots-attributes-1.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from twisted.web.template import Element, renderer, XMLFile

class ExampleElement(Element):
    loader = XMLFile('slots-attributes-1.xml')

    @renderer
    def person_profile(self, request, tag):
        # Note how convenient it is to pass these attributes in!
        tag.fillSlots(person_name='Luke',
                      profile_image_url='http://example.com/user.png')
        return tag
slots and attributes element - listings/slots_attributes_1.py
<div class="profile">
<img src="http://example.com/user.png" /> 
<p>Luke</p>
</div>
slots and attributes output - listings/slots-attributes-output.html

Iteration

Often, you will have a sequence of things, and want to render each of them, repeating a part of the template for each one. This can be done by cloning tag in your renderer:

<ul xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
    <li t:render="widgets"><t:slot name="widgetName"/></li>
</ul>
iteration template - listings/iteration-1.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

from twisted.web.template import Element, renderer, XMLFile, flattenString

class WidgetsElement(Element):
    loader = XMLFile('iteration-1.xml')

    widgetData = ['gadget', 'contraption', 'gizmo', 'doohickey']

    @renderer
    def widgets(self, request, tag):
        for widget in self.widgetData:
            yield tag.clone().fillSlots(widgetName=widget)

def printResult(result):
    print result

flattenString(None, WidgetsElement()).addCallback(printResult)
iteration element - listings/iteration-1.py
<ul>
    <li>gadget</li><li>contraption</li><li>gizmo</li><li>doohickey</li>
</ul>
iteration output - listings/iteration-output-1.xml

This renderer works because a renderer can return anything that can be rendered, not just tag. In this case, we define a generator, which returns a thing that is iterable. We also could have returned a list. Anything that is iterable will be rendered by twisted.web.template rendering each item in it. In this case, each item is a copy of the tag the renderer received, each filled with the name of a widget.

Sub-views

Another common pattern is to delegate the rendering logic for a small part of the page to a separate Element. For example, the widgets from the iteration example above might be more complicated to render. You can define an Element subclass which can render a single widget. The renderer method on the container can then yield instances of this new Element subclass.

1
2
3

<ul xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
    <li t:render="widgets"><span t:render="name" /></li>
</ul>
subview template - listings/subviews-1.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from twisted.web.template import (
    XMLFile, TagLoader, Element, renderer, flattenString)

class WidgetsElement(Element):
    loader = XMLFile('subviews-1.xml')

    widgetData = ['gadget', 'contraption', 'gizmo', 'doohickey']

    @renderer
    def widgets(self, request, tag):
        for widget in self.widgetData:
            yield WidgetElement(TagLoader(tag), widget)

class WidgetElement(Element):
    def __init__(self, loader, name):
        Element.__init__(self, loader)
        self._name = name

    @renderer
    def name(self, request, tag):
        return tag(self._name)

def printResult(result):
    print result

flattenString(None, WidgetsElement()).addCallback(printResult)
subview element - listings/subviews-1.py
<ul>
      <li><span>gadget</span></li><li><span>contraption</span></li><li><span>gizmo</span></li><li><span>doohickey</span></li>
</ul>
subview output - listings/subviews-output-1.xml

TagLoader lets the portion of the overall template related to widgets be re-used for WidgetElement, which is otherwise a normal Element subclass not much different from WidgetsElement. Notice that the name renderer on the span tag in this template is satisfied from WidgetElement, not WidgetsElement.

Transparent

Note how renderers, slots and attributes require you to specify a renderer on some outer HTML element. What if you don't want to be forced to add an element to your DOM just to drop some content into it? Maybe it messes with your layout, and you can't get it to work in IE with that extra div tag? Perhaps you need t:transparent, which allows you to drop some content in without any surrounding "container" tag. For example: 
<div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
<!-- layout decision - these things need to be *siblings* -->
<t:transparent t:render="renderer1" />
<t:transparent t:render="renderer2" />
</div>
transparent template - listings/transparent-1.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from twisted.web.template import Element, renderer, XMLFile

class ExampleElement(Element):
    loader = XMLFile('transparent-1.xml')

    @renderer
    def renderer1(self, request, tag):
        return tag("hello")

    @renderer
    def renderer2(self, request, tag):
        return tag("world")
transparent element - listings/transparent_element.py
<div>
<!-- layout decision - these things need to be *siblings* -->
hello
world
</div>
transparent rendering output - listings/transparent-output.html

Quoting

twisted.web.template will quote any strings that place into the DOM. This provides protection against XSS attacks, in addition to just generally making it easy to put arbitrary strings onto a web page, without worrying about what they might have in them. This can easily be demonstrated with an element using the same template from our earlier examples. Here's an element that returns some "special" characters in HTML ('<', '>', and '"', which is special in attribute values): 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from twisted.web.template import Element, renderer, XMLFile

class ExampleElement(Element):
    loader = XMLFile('template-1.xml')

    @renderer
    def header(self, request, tag):
        return tag('<<<Header>>>!')

    @renderer
    def footer(self, request, tag):
        return tag('>>>"Footer!"<<<', id='<"fun">')
renderers returning "special" characters - listings/quoting_element.py
Note that they are all safely quoted in the output, and will appear in a web browser just as you returned them from your Python method: 
<html>
<body>
  <div>&lt;&lt;&lt;Header&gt;&gt;&gt;!</div>
  <div id="content">
    <p>Content goes here.</p>
  </div>
  <div id="&lt;&quot;fun&quot;&gt;">&gt;&gt;&gt;"Footer!"&lt;&lt;&lt;</div>
</body>
</html>
output containing "special" characters - listings/quoting-output.html

Deferreds

Finally, a simple demonstration of Deferred support, the unique feature of twisted.web.template. Simply put, any renderer may return a Deferred which fires with some template content instead of the template content itself. As shown above, flattenString will return a Deferred that fires with the full content of the string. But if there's a lot of content, you might not want to wait before starting to send some of it to your HTTP client: for that case, you can use flatten. It's difficult to demonstrate this directly in a browser-based application; unless you insert very long delays before firing your Deferreds, it just looks like your browser is instantly displaying everything. Here's an example that just prints out some HTML template, with markers inserted for where certain events happen: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import sys
from twisted.web.template import XMLString, Element, renderer, flatten
from twisted.internet.defer import Deferred

sample = XMLString(
    """
    <div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
    Before waiting ...
    <span t:render="wait"></span>
    ... after waiting.
    </div>
    """)

class WaitForIt(Element):
    def __init__(self):
        Element.__init__(self, loader=sample)
        self.deferred = Deferred()

    @renderer
    def wait(self, request, tag):
        return self.deferred.addCallback(
            lambda aValue: tag("A value: " + repr(aValue)))

def done(ignore):
    print("[[[Deferred fired.]]]")

print('[[[Rendering the template.]]]')
it = WaitForIt()
flatten(None, it, sys.stdout.write).addCallback(done)
print('[[[In progress... now firing the Deferred.]]]')
it.deferred.callback("<value>")
print('[[[All done.]]]')
deferred example - listings/wait_for_it.py
If you run this example, you should get the following output: 
[[[Rendering the template.]]]
<div>
    Before waiting ...
    [[[In progress... now firing the Deferred.]]]
<span>A value: '&lt;value&gt;'</span>
    ... after waiting.
    </div>[[[Deferred fired.]]]
[[[All done.]]]
output from deferred example - listings/waited-for-it.html
This demonstrates that part of the output (everything up to "[[[In progress...") is written out immediately as it's rendered. But once it hits the Deferred, WaitForIt's rendering needs to pause until .callback(...) is called on that Deferred. You can see that no further output is produced until the message indicating that the Deferred is being fired is complete. By returning Deferreds and using flatten, you can avoid buffering large amounts of data.

A Brief Note on Formats and DOCTYPEs

The goal of twisted.web.template is to emit both valid HTML or XHTML. However, in order to get the maximally standards-compliant output format you desire, you have to know which one you want, and take a few simple steps to emit it correctly. Many browsers will probably work with most output if you ignore this section entirely, but the HTML specification recommends that you specify an appropriate DOCTYPE.

As a DOCTYPE declaration in your template would describe the template itself, rather than its output, it won't be included in your output. If you wish to annotate your template output with a DOCTYPE, you will have to write it to the browser out of band. One way to do this would be to simply do request.write('<!DOCTYPE html>\n') when you are ready to begin emitting your response. The same goes for an XML DOCTYPE declaration.

twisted.web.template will remove the xmlns attributes used to declare the http://twistedmatrix.com/ns/twisted.web.template/0.1 namespace, but it will not modify other namespace declaration attributes. Therefore if you wish to serialize in HTML format, you should not use other namespaces; if you wish to serialize to XML, feel free to insert any namespace declarations that are appropriate, and they will appear in your output.

Note: This relaxed approach is correct in many cases. However, in certain contexts - especially <script> and <style> tags - quoting rules differ in significant ways between HTML and XML, and between different browsers' parsers in HTML. If you want to generate dynamic content inside a script or stylesheet, the best option is to load the resource externally so you don't have to worry about quoting rules. The second best option is to strictly configure your content-types and DOCTYPE declarations for XML, whose quoting rules are simple and compatible with the approach that twisted.web.template takes. And, please remember: regardless of how you put it there, any user input placed inside a <script> or <style> tag is a potential security issue.

A Bit of History

Those of you who used Divmod Nevow may notice some similarities. twisted.web.template is in fact derived from the latest version of Nevow, but includes only the latest components from Nevow's rendering pipeline, and does not have any of the legacy compatibility layers that Nevow grew over time. This should make using twisted.web.template a similar experience for many long-time users of Twisted who have previously used Nevow for its twisted-friendly templating, but more straightforward for new users.

Index

Version: 12.0.0