Hi,<div><br></div><div>I have an application that fetches many urls, following any redirects. Every so often I get a variety of unhandled exceptions from within twisted.web.client (see snippet at the bottom of email).</div>
<div><br></div><div>My code has an addErrback attached to the deferred and a try/except around the getPage() call, neither of which get invoked.</div><div><br></div><div>Are these exceptions that web.client needs to catch internally and then call self.deferred.errback so that my error handler will receive it?</div>
<div><br></div><div>If not, how can I catch them?</div><div><br></div><div>Cheers,</div><div>Chris</div><div><br></div><div>--- Exception 1 ---</div><div><br></div><div><div>2009-10-06 20:21:16-0700 [HTTPPageGetter,client] Unhandled Error</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>Traceback (most recent call last):</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 84, in callWithLogger</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return callWithContext({"system": lp}, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 69, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return context.call({ILogContext: newCtx}, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 59, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.currentContext().callWithContext(ctx, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 37, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return func(*args,**kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>--- <exception caught here> ---</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = getattr(selectable, method)()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 463, in doRead</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.protocol.dataReceived(data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 231, in dataReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = self.lineReceived(line)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 420, in lineReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> self.handleEndHeaders()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 74, in handleEndHeaders</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> m()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 93, in handleStatus_301</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> _parse(url, defaultPort=self.transport.getPeer().port)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 426, in _parse</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> port = int(port)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>exceptions.ValueError: invalid literal for int() with base 10: ''</div>
<div><br></div><div>--- Exception 2 ---</div><div><br></div><div><div>2009-10-06 20:28:42-0700 [HTTPPageGetter,client] Unhandled Error</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Traceback (most recent call last):</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 84, in callWithLogger</div><div>
<span class="Apple-tab-span" style="white-space:pre"> </span> return callWithContext({"system": lp}, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 69, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return context.call({ILogContext: newCtx}, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 59, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.currentContext().callWithContext(ctx, func, *args, **kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 37, in callWithContext</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return func(*args,**kw)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>--- <exception caught here> ---</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = getattr(selectable, method)()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 463, in doRead</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.protocol.dataReceived(data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 231, in dataReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = self.lineReceived(line)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 420, in lineReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> self.handleEndHeaders()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 74, in handleEndHeaders</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> m()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 93, in handleStatus_301</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> _parse(url, defaultPort=self.transport.getPeer().port)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 425, in _parse</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> host, port = host.split(':')</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>exceptions.ValueError: too many values to unpack</div>
<div><br></div><div>--- Exception 3 ---</div><div><br></div><div><div>2009-10-06 20:31:11-0700 [HTTPPageGetter,client] Unhandled Error</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Traceback (most recent call last):</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 84, in callWithLogger</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> return callWithContext({"system": lp}, func, *args, **kw)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 69, in callWithContext</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> return context.call({ILogContext: newCtx}, func, *args, **kw)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.currentContext().callWithContext(ctx, func, *args, **kw)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> return func(*args,**kw)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>--- <exception caught here> ---</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = getattr(selectable, method)()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/internet/tcp.py", line 463, in doRead</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> return self.protocol.dataReceived(data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/protocols/basic.py", line 231, in dataReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> why = self.lineReceived(line)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line 404, in lineReceived</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> status = l[1]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>exceptions.IndexError: list index out of range</div><div><br></div>
</div></div></div>