[Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot
Craig Rodrigues
rodrigc at crodrigues.org
Mon Jan 23 17:08:57 MST 2017
On Sat, Jan 21, 2017 at 8:06 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
wrote:
>
> On Jan 21, 2017, at 6:15 PM, Craig Rodrigues <rodrigc at crodrigues.org>
> wrote:
>
> If I run the test on Python 2, I don't get the error, and on line 93,
> brdicts is a dict.
> However, if I run the test on Python 3, brdicts is a Deferred, thus the
> error.
>
>
> This really ought to be impossible. If I were seeing this,
> single-stepping through inlineCallbacks in pudb would probably be my next
> step. Have you made any attempts to simplify it down to remove some of the
> buildbot-specific code?
>
> -glyph
>
>
I did some more debugging. The callstack is quite deep. :/
I used this command to trigger the error:
trial
buildbot.test.unit.test_process_buildrequestdistributor.TestMaybeStartBuilds.test_slow_db
I found in this function in buildbot's BuildRequestsEndpoint.get() function
(
https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146
) there is this line:
* defer.returnValue( [(yield self.db2data(br)) for br in
buildrequests])*
On Python 2, this line returns:
an object list
each list entry is a dictionary of name/value pairs that looks like:
[{'buildrequestid': 10, 'complete': False, 'waited_for': False,
'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11,
'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6,
40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for':
False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid':
11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13,
30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
'priority': 0}]
On Python 3, this returns:
an object <generator object BuildRequestsEndpoint.get.<locals>.<listcomp>
of type <class 'generator'>
The self.db2data() function returns defer.succeed(data) (
https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L32
)
So it looks like on Python 3, this is not getting evaluated properly, and
the upper layer is trying
to index a Deferred instead of a list of dictionaries.
I'm not so familiar with generators/Deferreds in Py2 vs. Py3. What is the
best way
to fix this to get consistent behavior on Py2 and Py3?
Thanks.
--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170123/63885a1d/attachment-0002.html>
More information about the Twisted-Python
mailing list