[Twisted-Python] Scaling problem

[Tommi Virtanen]

Uwe C. Schroeder wrote:
> is there a way to have one service start several processes ?
> When using twisted everyone connecting to the server ends in the same process. 
> This leads to "slowing down" when requesting information whenever a certain 
> number of people go on. Each request takes some time to process and meanwhile 
> the server is less responsive. I have one particularly expensive function 
> that takes a bit. So if several people happen to use this function at a time 
> the server slows down, up to where it takes a minute to complete.
> It's not a problem of system ressources (it's a quad processor xeon server 
> with plenty of RAM and ultra 320 scsi drives running only this application 
> and the postgres backend). The machine has a load of 0.2 while slowing down.
> Basically most of the machine is idle, since it only runs 2 processes and has 
> 4 processors.

You should look at having a pool of worker _processes_, to which you 
dispatch work with pb.

Not threads, as python threading will not really scale, and there's 
really no point in threading if the amount of input/output to the worker 
is small enough (shared mem has its good and bad sides).

> Is there a way to have twisted do something like apache or postgres and split 
> the incoming connection to several processes ? It's not a web application, 
> I'm solely using pb.

If you listen in a parent process, fork multiple children, and make the 
children accept, unix does the right thing. I don't think Twisted knows 
how to do that out of the box, but it should be doable.

That's assuming you don't need (much) shared data between the children.

> BTW: You might suggest to break down the expensive function into smaller 
> pieces - no way. It has to run in one database transaction otherwise I might 
> end up with a inconsistent database - no good in finance business.

If you are lucky, you can find a threading DB library (adbapi) that 
actually scales. The other mail in this thread are more helpful there..