[Twisted-Python] Simultaneously client-server app

[Thomas Weholt ( PRIVAT )]

I have some questions which seem to be related to this.

I have a webserver which serves as a newsaggregator etc. Once a while a
"worker" ( being a thread or not ) must go out and collect, parse and store
new items served by rss/rdf-feeds from around the net. Using a
SQLite-database which takes one connect at the time, at least as I
understand it, using threads could pose a serious problem if do-able at all.
I have another "worker" collecting information about files in given folders,
generating sha1-checksums etc. which is *very*  time consuming on large
amounts of data. This info must also go into the database.

So far I got the code that schedules a call and re-schedules it after the
call has finished. But the method that's called is blocking and has to be
re-written using deferreds or threads or something.

<dream-mode value="on" requirements="stupidity filter">
The utlimate goal would be to have some way of letting the user easily and
quickly define new workers doing user-defined things without hoggin the
server and by using as little as possible of strange, weird, technical,
complicated Twisted-things like deferreds, callbacks, threads etc.

My question is; is there a generic way to make old-style blocking Python
code twisted-friendly non-blocking using deferToThread or deferreds wrapped
around it?
</dream-mode>

I'm going to look into tnntprss.py which was mentioned yesterday or
something on the lists, which does some of the things I need, but any
example on how to go about this would be nice.

PS! Another question; should these workers be located in an
ApplicationService-subclass?

Thomas



----- Original Message -----
From: "Andrew Bennetts" <andrew-twisted at puzzling.org>
To: <twisted-python at twistedmatrix.com>
Sent: Friday, July 11, 2003 9:52 AM
Subject: Re: [Twisted-Python] Simultaneously client-server app


> On Fri, Jul 11, 2003 at 04:55:23PM +1000, Minh Luu wrote:
> >    Twisted Masters,
> >
> >    Has anyone any suggestion on how I might start on the following task
using
> >    Twisted?
> >
> >    What I would like to do is a type of server that will serve a number
of
> >    clients and itself is a client of other servers. It requires a spool
of
> >    worker threads to perform time-consuming tasks and database related
> >    tasks.  All the network I/O will be handled by a single async thread
using
> >    select. A simple diagram below demo the model I would like to
implement.
> [snip pretty diagram]
>
> I doubt threads and select are actually requirements you have, rather,
it's
> just how you think it should work.  But if you know how it should work,
why
> ask us?  ;)
>
> There are lots of blanks in your description, so it's hard to guess what
you
> need to know.  You effectively have asked a couple of very general
> questions, so I'll give some general answers, and hope that's enough.  If
it
> isn't, try being more specific about what you're doing, and I'll try to
give
> more specific answers.
>
>
> * Question: *
> "What I would like to do is a type of server that will serve a number of
> clients and itself is a client of other servers."
>
> * Answer: *
> Network clients and servers in Twisted are pretty easy, and its very good
> at them -- have you seen these HOWTOs?
>     http://twistedmatrix.com/documents/howto/servers
>     http://twistedmatrix.com/documents/howto/clients
>
> Being a server and a client at once is as easy as:
>     reactor.listenTCP(1234, MyServerFactory())
>     reactor.connectTCP('host', 4321, MyClientFactory())
>
>
> * Question: *
> "...a spool of worker threads to perform time-consuming tasks..."
>
> * Answer: *
> Have you seen the "Using Threads" HOWTO?
>     http://twistedmatrix.com/documents/howto/threading
>
> The key tool you probably want to use when working with threads is
> twisted.internet.threads.deferToThread.
>
> Also, be aware that threads aren't necessarily the only solution to
> "time-consuming tasks", although it would help to know what these tasks
> actually are :)
>
>
> * Question: *
> "...and database related tasks"
>
> * Answer: *
> The twisted.enterprise package has support for using standard Python
DB-API
> 2.0 compliant modules.  You probably just want to use
> twisted.enterprise.adbapi, which is described here:
>     http://twistedmatrix.com/documents/howto/enterprise
>
> That document is a little out of date, unfortunately -- it doesn't mention
> that you can avoid adbapi.Augmentation if you don't like that approach,
and
> just use adbapi.ConnectionPool directly, although the its API is fairly
ugly
> and not very Twisted-like.  That module could probably do with an
overhaul,
> but that's not your problem ;)
>
>
> I hope I've helped at least a little.
>
> Regards,
>
> -Andrew.
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>