[Twisted-Python] One big smile ...

[Thomas Weholt]

I got a list of RSS/RDF-urls and I iterate thru them in sequence, feeding
each one to Mark Pilgrims Ultra Liberal Parser. The call to that
parse-method is actually what's being using in deferToThread. The I add a
callback-method which processes the result of the RSS/RDF-parsing. It's a
plain dictionary. This dictionary is processed into a simpler list. That's
it.

Alot of the things I'll be doing is like this; fetch a file from the net (
here the parser fetches it itself ), process the data returned from that
call and update some object or stuff the result into a database. The last
option most of the times probably, and the database will most likely be
SQLite which is not very thread-frinedly either. Other times it can be a
XML-RPC/SOAP-call which might take time to complete and the result of the
call might have to be processed. The last thing I can think of is scanning
folders and generating checksums. Probably the worst job of them all. Not
done often, but very time/resource-consuming when it occurs.

Most often I'll have to update something in the main thread, some state or
value, so if there's any way to use mutex/locks/whatever or ways to avoid
the whole thing I'm all ears ( one of the worlds most moronic statements
BTW ).

Thomas



----- Original Message -----
From: "Christopher Armstrong" <radix at twistedmatrix.com>
To: <twisted-python at twistedmatrix.com>
Sent: Saturday, July 12, 2003 2:29 AM
Subject: Re: [Twisted-Python] One big smile ...


> On Sat, Jul 12, 2003 at 02:05:16AM +0200, Thomas Weholt wrote:
> > I implemented a small newsfetching webserver today. At first I used some
> > blocking code at startup and then reactor.callLater(some_intervall,
> > sameBlockingMethod) to call it later and refresh the content later. It
> > processed about 8-10 RSS/RDF feeds and then handed my server back to me
so I
> > could see the output. I was quite happy about the result. Just for fun I
> > tried to do the blocking code in a thread instead, - with mixed results.
> > Read some more docs. And did a deferToThread for each call to the
blocking
> > method. Added a callback-method that parsed the results. A bit more
hacking
> > and ...then .. suddenly ... it was just .... so fast. No hogging the
server
> > at startup, all content fetched,, *much* faster than before. Everything
> > seemed great. I had finally got the grip on threads and deferred. But
> > something was wrong with my data. Some data were in the wrong lists etc.
so
> > I guess I need some mutex-locking or something. This is were you guys
come
> > in. Do I need to aquire a lock to avoid updating wrong parts of my
> > datastructure?
>
> Why are you even using threads? What API are you using that is only
> available as blocking calls?
>
> --
>  Twisted | Christopher Armstrong: International Man of Twistery
>   Radix  |          Release Manager,  Twisted Project
> ---------+     http://twistedmatrix.com/users/radix.twistd/
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>