[Twisted-Python] Scaling problem

[Itamar Shtull-Trauring]

On Wed, 5 Nov 2003 02:48:07 -0800
"Uwe C. Schroeder" <uwe at oss4u.com> wrote:

> I think I suffer from database latency in general and with the
> expensive function in particular. The expensive function could be
> moved to a thread and maybe I'll give it a try. However if I do this
> for every method that writes more than 10 or 15 records to the
> database I'll most likely end up with thousands of threads.

No, Twisted has a thread pool, so if you are using Twisted APIs you
won't get more than the max number of threads the pool is configured to
use.

> The most expensive function I have (besides the printing mentioned
> above) executes in about 3 seconds. This is not "blocking" in my
> understanding, since the function never waits for something to happen,
> it just calculates and stores stuff. Twisted is unresponsive for those
> 3 seconds.

If Twisted is unresponsive, that means it blocks. Any function that
takes more than 10ms should probably count as blocking, really. Try
running it in a thread. Because of the global interpreter lock, this
won't reduce runtime, but it'll enhance responsiveness to other clients.

In general, it sounds like you are doing a lot of database queries. Some
ways to optimize this:

1. Do less queries - cache, merge identical requests, etc.. 
2. Do a number of queries in a single transaction in a single thread,
rather than deferToThread, get result, run another query with
deferToThread again, etc..
(3. Don't use RDBMS. Use an in-process database like ZODB or Quotient's
atop when it's done.)

-- 
Itamar Shtull-Trauring    http://itamarst.org/
Available for Python & Twisted consulting