[Twisted-Python] App design / Twisted Logging?

[Henning.Ramm at mediapro-gmbh.de]

Paul, thank you very much for taking the time to answer my confusion.

>> -> Do I need a thread for every copy/move operation (large 
>files via network)?
>
>Nope. That's the beauty of twisted code. I've written this 
>sort of handling
>multiple packet trains many times. Write a protocol.

Hm, sounds like using FTP and not shutil.move via system's SMB?
Or is there already something better in twisted?

If I'd write my own protocol to wrap shutil.move,
the operation would be blocking because I don't
work on file block level and the protocol doesn't make sense.
I'm not feeling like writing file transfer code on file block level.
Or do you think it would be worth the hassle?

>> Another part reads some XML files (using minidom) and writes 
...
>I ran into long processing problems before. There's a simple 
>solution. Pick
>something convenient such as simply chunking everything up 
>into processing one line at a time.

So not using DOM but per-line XML handling.
Not that convenient, because I use only small bits
of all the XML data that are spread all over the files,
but I guess it would become better twisted code.

>your constructs using deferreds. And the Flow module allows 
>you to go even
>further than the hard coded version I'm suggesting here. Try 
>reading through
>the Flow module documentation. Just don't get hung up about 
>using it. For my
>purposes, it was usually better just to have the basic concept 
>in mind than to actually use Flow.

That's perhaps a general problem with twisted: There are great
solutions for everything, but you need to know them in detail
to know which fits your problem. Or you must know how you should
reshape your problem to fit in some twisted solution...

I know twisted "does it all"(TM), but was is "it"? ;-)


>Yep. Frequently, this is where I'd run a deferred() just after 
>sending the request to the known-to-be-slow process.
>Read that section carefully too. Then
>your code gets called back immediately when the process 
>completes. You will
>need to understand deferred's to fully utilize twisted's 
>threading interface,
>and then you'll realize that you probably don't need threads.

I'm just trying to write a simple directory watcher (I need
this at every corner of my app), Patrick Lauber wrote an answer
to my initial question on that, but that wasn't really what I needed.

It works so far that it calls a deferred callback if it gets a
notify on a new/changed file, but only once; next time I get an
"AlreadyCalledError" - looks like I don't yet understand deferreds.

At the moment it's inherited from pb.Root, because I'll need it to
run remotely sometimes, but perhaps it would be better to use a
service or something else -- it should run "all the time" if not
stopped and call a callback for every file.
I attached the file, perhaps someone can point out my biggest mistakes?


>first but once you get used to it, deferred's seem just well...obvious.

I hope to get into that higher state of mind soon. ;-)

>As to logging itself...it might be easier to just write your 
>own. Usually I
>haven't been overly thrilled with the mechanical-looking output of
>off-the-shelf loggers and logging is little more than just 
>printing to a file
>anyways, sometimes with some extra (chop it up and reclaim 
>space) features added.

I enjoyed being able to switch the logging output, e.g. from file to
database or email per config file without the need to go into the code.
I don't feel like re-inventing the wheel, but as Glyph pointed out,
the config syntax of standard logging is just ugly and messy; the
config syntax of log4[j|perl|net] ist much more logical. Perhaps
someone should write a log4twisted module...


Best regards,
Henning Hraban Ramm
Südkurier Medienhaus / MediaPro
Support/Admin/Development Dept.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dirwatcher.py
Type: application/octet-stream
Size: 2570 bytes
Desc: dirwatcher.py
URL: </pipermail/twisted-python/attachments/20050311/1e803d04/attachment-0002.obj>