[Twisted-web] Twisted & legacy code - Unorthodox? Opinions wanted

[Harald.Nyman at astrazeneca.com]

Apologies for a long post. Felt it was needed to get the message across.

What's described could be a creative approach to dealing with legacy code,
but it could also be an ugly hack, and I would really benefit from the views
of you gurus as to whether this looks like a good/acceptable/bad way to use
Twisted. (If 'bad' - any immediate suggestions? :))

Thanks in advance,
Harald


We have an existing web app (~10 000 lines of Python, generating HTML, with
25 cgi scripts), running on Apache. We are considering Twisted as one
alternative to make it faster and better structured. Just running the
existing code with Twisted instead of Apache doesn't increase performance
much, as CGI's process startups are the bottleneck. So, to really gain from
Twisted, I guess we need to make the code event-driven (and this would be
the ultimate goal).

Now, one design has been proposed, which boosts performance dramatically
with few code changes. It uses Twisted as a plain TCP socket server,
communicating with text strings instead of using HTTP. The browser-server
interaction is -
 - Request: "scriptSelectorKeyword::formInputDict::cookiesDict"
 - Response: "generatedHTML" (as before, but with no line breaks)

Creating the request in the browser:
All links in the page are modified to NOT trigger a GET/POST, but use
javascript to
 - scan form input (<INPUT>, <SELECT>, etc) and build formInputDict ("{name:
value, ...}")
 - create a similar representation of the cookies, as cookiesDict
 - send the "::"-delimited string using a small applet

The server (having imported all modules used by the app at startup) -
 - evals formInputDict and cookiesDict (since they come as text strings)
 - based on scriptSelectorKeyword, invokes the appropriate module in a
thread (threads.deferToThread)
 - (we don't use CGI, so formInputDict and cookiesDict replace Python's cgi
& Cookies modules)
 - the module has a "getBuffer" method that returns the generated HTML
instead of printing it
 - the generated HTML is written to the socket
(reactor.callFromThread(self.transport.write, html))

The browser
 - receives the generated HTML string through the applet
 - parses the cookies
 - displays the HTML, using document.write

This design is up and running with impressive performance, and the users are
enthusiastic. However, the maintenance team is less so, and these are our
main concerns (with the advocates' answers added) -

Is this really a valid way to use Twisted?
 - Think so. The docs say, "Twisted still allows the use of threads if you
really need them, usually to interface with synchronous legacy code", but we
should ask the experts.

Will it scale? Will it work with future Twisted versions?
 - Hope so. See previous point.

It's not a web app anymore, it's a javascript app.
 - So? 

It locks us into MS Internet Explorer.
 - Yes. However, MSIE is our corporate standard anyway.

It can't be used through a firewall.
 - No. Luckily, this is a strictly internal app.

We don't want to rely on and maintain home-written javascript.
 - Good point. However,
   1) We're talking about roughly 2 A4 pages
   2) The javascript to handle form input & cookies is meant to be general,
and once it's validated, we should not need to touch it much.
   3) If we accept being tied in with MSIE, ain't its javascript part of the
package?

We can't do unit tests on the javascript.
 - Correct. See previous point.