[Twisted-Python] How to use ampoule?

[Valentino Volonghi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Feb 20, 2009, at 2:08 AM, Chris wrote:

> Hi everyone,
> I am using twisted to build a server,and the computing request maybe
> costs lot of cpu resources.I have checked the maillist,it seems I can
> use ampoule plugin to create another process ,I have checked the
> website,and downloaded the source code.It seems there is no any  
> document
> for it.I did find a example dir,after checking the code,I was
> confused.There should be two files,one for client and one for  
> server,am
> I right?the client will send the request to the server(could be in
> another machine?),and the server responses.but I can't find anything
> which matches what I thought.Can anyone explain a little bit to me? 
> Or if
> there is some code that would be better.

I don't have much time to write documentation, I basically spend most of
the time in documenting the code itself, in testing it and I attached  
some of
the examples that one can use to learn the basics by himself but...

You don't need 2 files, you need the code you want to run on both sides
of the connection, Ampoule doesn't support shipping functions, although
one might implement it on top of what ampoule offers, albeit not  
recommended.

AMPoule uses AMP as a communication protocol between the caller and the
process pool, this means that they talk to each other using  
twisted.protocols.amp
and that for the abstraction sake they work as 2 separate networked  
services
that make RPC calls to each other. Again this also means that in order  
to
develop a process pool you will use AMP abstractions and classes on top
of what AMPoule offers by default.

Let's look at the simplest example: examples/pid.py

What you need is a set of commands that a child process should be able
to answer to, in pid.py this set is made of just a single command and  
that
is:

from twisted.protocols import amp

class Pid(amp.Command):
     response = [("pid", amp.Integer())]

Once you define a command that you want to be able to run in a process
pool you need to define a child process that is able to answer to the  
Pid
command. Defining this also defines what every worker in the process
pool will be able to answer to.

In the example this is done with the following lines:

from ampoule import child

class MyChild(child.AMPChild):
     @Pid.responder
     def pid(self):
         import os
         return {"pid": os.getpid()}

We define a child of the process pool called MyChild and using AMP
machinery we set the method pid as the responder for the command
Pid. Unsurprisingly this command gets the pid and returns it.

Now we need to run this code and use it. So far we have defined
the server (ProcessPool) side of things.

[NOTE: In order to run everything in a single file using "python  
filename.py"
to start it we need to hack around python's import system, this is why
util.mainpoint exist, only to allow the script to import itself,  
you'll notice
that a script's name when started with "python filename.py" is not
filename but __main__.]

Here's the code

@util.mainpoint
def main(args):
     import sys
     from twisted.internet import reactor, defer
     from twisted.python import log
     log.startLogging(sys.stdout)

     from ampoule import pool

     @defer.inlineCallbacks
     def _run():
         pp = pool.ProcessPool(MyChild, min=1, max=1)
         yield pp.start()
         result = yield pp.doWork(Pid)
         print "The Child process PID is:", result['pid']
         yield pp.stop()
         reactor.stop()

     reactor.callLater(1, _run)
     reactor.run()

Besides all the standard twisted imports and 'boilerplate' code
the core of the client is inside the _run function.

It creates the process pool telling it to use MyChild as a specification
for its children, we also tell it that the minimum size of the pool is 1
as well as the maximum size of the pool. Then the code proceeds
to start it and once it's tarted we can submit commands and this is
done with:

result = yield pp.doWork(Pid)

or

result = yield pp.callRemote(Pid)

The script then prints the result, stops the pool and exits.

Strictly speaking the client of the process pool only needs to know
the commands that he wants to execute, and which pool (if there
is more than one) can execute it.

This however is only true when the server is started separately from
the client using "twistd ampoule" plugin. If your clients starts the
pool by itself it needs to know also the class that defines the children
protocol.

The twistd ampoule plugin does nothing more than taking a child and
parent class (the parent class defines what the master of the process
pool can speak so that children can make calls against it if needed)
and using some hacks exposes the same exact interface across the
network using the parameters that you pass in the cli.

> And the twisted server will receive lot of binary data from the
> client,if I use ampoule,I have to send the same data to the ampoule
> server again(now the twisted server acts as the ampoule client).Is the

Correct. I don't see any other way to do this except maybe if you have
a distributed filesystem, in which case you might want to just save
the data on the server and then make the right calls on the remote
pools that have access to this distributed filesystem so that they can
process the shards that they have access to.

> ampoule suitable for such a kind of task? Or I should just use twisted
> process module?

Using the process module is not really different, the only issue you
might find is related to the limit imposed by AMP of 64KB of data
that can be transmitted in a single call to a given child. So for  
example

pool.callRemote(Command, argument="I'm a bit string of more than 64KB")

won't work but:

pool.callRemote(Command, argument="I'm a bit string of LESS than 64KB")

will. There are multiple ways in which you can solve this issue but  
essentially
I wouldn't use ampoule to transport big quantities of data, you'd  
better use
a distributed filesystem or actually an http server where you store data
from the server and each worker grabs data from.

> I don't know my understanding is correct or not,please correct me.


It seems you got it right.

HTH

- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkmfN88ACgkQ9Llz28widGXE9ACgk2OAlXK0cVP5/5tINoFAD70C
Zc8Anig2L8GCNklG83a6la4x/hksFozW
=SS7Z
-----END PGP SIGNATURE-----