[Twisted-web] how to send parallel requests to the dbms while rendering html with Cheetah

[Jean-Paul Calderone]

On Mon, 23 Jan 2006 01:03:42 +0100, Andrea Arcangeli <andrea at cpushare.com> wrote:
>Hello,
>
>With Cheetah I have full control on the serialization or parallelism of
>my queries and thanks to a real dbms I can execute very expensive
>queries in parallel on multiple cpus with just a small change:
>
> [snip big diff]
>
>I've an old (slow) 2-way smp server, and by sending (readonly) queries
>to postgresql in parallel I just reduce the time it takes to render the
>usual klive pages from 2.9sec to 1.49sec (2 cpus working at 100%
>executes the queries in parallel and so the results now arrives in
>_exactly_ half the time, postgresql is really scaling perfectly in this
>test). This new smp improvement comes after the huge boost I had after
>moving from Nevow to Cheetah for the rendering of the html (from >5.9sec
>to 2.9sec). On my 4-way dualcore opteron desktop the rendering time
>changes from ~1sec to ~370msec.
>
>So the overall since I started migrating to Cheetah, the time it takes
>to render the klive homepage went down from 5.9sec (old
>nevow+twisted.web code) to 1.5sec (new cheetah+twisted.web2 code). A
>~300% performance improvement in rendering pages isn't bad. The code is
>almost unchanged, just the html rendering model has changed.
>
>This is just to share my real life experience and perhaps this could be
>useful to others.
>
>Now the 1.5sec are almost all spent waiting the sql queries and anyway
>both cpus are fully utilized (Cheetah renders the first #blocks while
>the other cpu calculates the results for the next block in parallel), so
>there's probably not much left to optimize (nothing as easy as the
>changes I did in the last days to move from 5.9sec to 1.5sec at least).
>

That's great, Andrea.

>Parallel expensive queries to the db, is something that the axiom model
>will never be able to achieve (as long as the api isn't changed to
>return deferreds to allow real dbms instead of sqllite, which means all
>code written today for the current axiom API will have to changed
>significantly). Having to schedule to another context while waiting the
>deferred to fire, may open a window for for race conditions too,
>and have fun writing unit-test verifying race conditions.
>
>While you can scale the network load by running multiple twisted
>servers, you can't scale the db queries coming from different twisted
>servers in parallel with the axiom API. This ignoring the fact nevow
>doesn't contemplates the possibility of sending all queries in parallel.

I don't understand why you insist on continuing to harp on this point, though.  Even if it were accurate (which I must stress, it /isn't/, and if you care to discuss the technical merits of Axiom, I'd be happy to do so on <divmod-dev at divmod.org> but /not/ here), no one cares.  You've been asked repeatedly to let it go, and by refusing to do so you are being very rude to the people who wrote the software you continue to use.  Please, you do your thing, let us do ours.

Jean-Paul