[Twisted-Python] Twistd thread problem

[Jan Van Uytven]

Hi all,

I coded an SMTP proxy that receives incoming e-mail requests and pushes 
the received e-mail onto a Queue were it is processed by a worker thread 
that finishes by sending the e-mail on to another SMTP server.

The server part of the proxy was written in Twisted, but the worker 
thread uses smtplib to fire off the processed e-mails (I didn't know how 
to use the twisted smtp client in a thread).

When I run the application using twistd -noy <appname>, it works great. 
When I try to daemonize it (twistd -oy <appname>) the server part works, 
but the worker thread doesn't. It creates and then I suspect it 
immediately dies, as no error is returned upon creation. A debug of the 
queue reveals that the thread is not pulling items off it. A debugging 
statement placed at the top of the thread never runs, and the twistd log 
reports nothing wrong. I'm using reactor.callInThread to start the 
worker function.

I tried running twistd --spew -oy <appname> and captured the log it 
produced. I did a filter for anything with 'thread' in it, this is the 
result:

2005/05/16 09:15 PDT [-] function ? in 
/usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 11
2005/05/16 09:15 PDT [-] function WorkerStop in 
/usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 23
2005/05/16 09:15 PDT [-] function init in 
/usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 219
2005/05/16 09:15 PDT [-] function ThreadPool in 
/usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 31
2005/05/16 09:15 PDT [-] function ThreadSafeList in 
/usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 199
2005/05/16 09:15 PDT [-] function init in 
/usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 219
2005/05/16 09:15 PDT [-] method __init__ of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method start of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method adjustPoolsize of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method _startSomeWorkers of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method callInThread of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method _startSomeWorkers of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method startAWorker of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] method __init__ of threading.Thread at -1216568116
2005/05/16 09:15 PDT [-] method __init__ of threading.Thread at -1216568116
2005/05/16 09:15 PDT [-] method _set_daemon of threading.Thread at 
-1216568116
2005/05/16 09:15 PDT [-] function currentThread in 
/usr/lib/python2.3/threading.py, line 669
2005/05/16 09:15 PDT [-] method isDaemon of threading._MainThread at 
-1214935060
2005/05/16 09:15 PDT [-] function Condition in 
/usr/lib/python2.3/threading.py, line 147
2005/05/16 09:15 PDT [-] method __init__ of threading._Condition at 
-1216567604
2005/05/16 09:15 PDT [-] method __init__ of threading._Condition at 
-1216567604
2005/05/16 09:15 PDT [-] method start of threading.Thread at -1216568116
2005/05/16 09:15 PDT [-] method _note of threading.Thread at -1216568116
2005/05/16 09:15 PDT [-] method run of threading.Thread at -1216568116
2005/05/16 09:15 PDT [-] method _worker of 
twisted.python.threadpool.ThreadPool at -1216569236
2005/05/16 09:15 PDT [-] function currentThread in 
/usr/lib/python2.3/threading.py, line 669
2005/05/16 09:15 PDT [-] function threadMailer in twisted-copyuser.py, 
line 201
2005/05/16 09:15 PDT [-] method acquire of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] method release of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] method acquire of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] method release of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] method acquire of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] method release of 
twisted.python.threadable._XLock at -1216639284
2005/05/16 09:15 PDT [-] function registerAsIOThread in 
/usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 262

Why would the worker thread act differently with twistd daemonized? Any 
suggestions, advice, or hints would be welcome.

Thanks,

Jan

-- 
Jan Van Uytven
Computer Systems & Network Engineer
Convergent Media Network
wyvern at crm3.com

Only in silence the word,
only in dark the light,
only in dying life:
Bright the hawk's flight 
on the empty sky.

- Ursula K. LeGuin, 'A Wizard of Earthsea'

*** LEGAL STUFF ***
This electronic message transmission, including any attached files, 
contains information which may be confidential and/or privileged.
The information is intended to be for the SOLE use of the individual or 
entity named above. If you are not the intended recipient, be aware 
that any review, disclosure, copying, distribution or use of the 
contents of this information is prohibited. This message is not to be 
forwarded for any reason without prior consent of the sender. 

If you have received this electronic transmission in error, please 
notify the sender immediately by telephone (250) 386-4266 or by a 
'reply to sender' only message and destroy all electronic and hard 
copies of the communication, including attachments, without reading 
or saving in any manner.