[Twisted-Python] Python 3 support in Twisted: Request for relaxation of coverage requirements

[Vladimir Perić]

Hello everyone,

as some of you may be aware, I was accepted as a Google Summer of Code
student, with the project of porting Twisted to Python 3 (or getting
as close as possible)[1]. Now, there are basically two strategies:
maintaining py2 code and using 2to3 when installing under py3, or
maintaining a codebase which is compatible with both 2 and 3 (a third
strategy is to have two codebases, but I feel this would introduce a
too high maintenance burden).[2] What both these approaches have in
common, is that it is advantageous (and mandatory if not using 2to3)
to update the syntax to more modern idioms (eg. exception syntax, uses
of apply and has_key, the print statement...). And therein lies my
problem: Twisted policy is that all code should be tested, and tests
should be added to modified, untested code... and I have to make a lot
of trivial syntax changes to a lot of code, some of which is untested.
Now, I agree that tests are very important, and in general I'm for the
100% coverage policy Twisted is trying to approach, but I feel that
this is beyond the scope of my project and, more importantly, would
slow my progress considerably.

As such, I would like to request permission for the coverage
requirements to be relaxed for specific Python 3 changes. As mentioned
above, this would allow me to progress much faster and concentrate on
the bigger incompatibilities. Of course, I would still write tests for
more complex code, if and where required. Furthermore, I propose to
use the 2to3 tool to make these conversions, which relies on analysis
of the underlying code - as such, it is much better than any
grep-related solution and much less prone to errors. It is also very
widely used, so I consider it a very robust solution. Still, I realize
this would be quite a big breach of policy, so I would like to make a
case for each change individually. This list is not exhaustive, merely
what I have encountered so far.

1) print statement

Probably the most noticeable difference, the print statement is a
function in Python 3. Now, if we opt for a single, compatible
codebase, all print statements should be converted to functions.
Luckily, 2.6 can import it from __future__ so this is not a problem.
There's more than 1000 print statements in Twisted code and even just
checking if each is covered would be a daunting job. Even if no other
exception is permitted, I think this one makes a lot of sense as it's
very hard to do anything wrong here.

2) has_key method

The issue that started this (see ticket 4053); the has_key method on
dicts has been changed to the builtin keyword "in" ("key in dict"
instead of dict.has_key(key)). Now, exarkun disliked my automatic
change, so I split the work into tested and untested uses (the tested
are fixed) but there's still about a hundred remaining. Now, a concern
here is that some Twisted classes could also be defining has_key
methods, but all three such classes also provide an equivalent
__contains__ method so this cannot be a problem (plus, I believe the
has_key methods should be deprecated but that's a separate issue). In
the end, this is a purely syntatic change and I do not feel like
writing tests everywhere around Twisted.

3) apply

To quote Lennart Regebro, "The Python 2 builtin apply() has been
removed in Python 3. It’s used to call a function, but since you can
call the function directly it serves no purpose and has been
deprecated since Python 2.3." I've replaced those covered by tests,
but there's still some 23 untested uses left, most of them in
spread/ui/tkutil, and I would *really* prefer not to write tests for
that.

4) Exceptions

The exceptions syntax has changed to be less ambiguous, and the new
"except Exception as e" syntax is available in 2.6 and up. This is a
trivial syntax change (change a comma to "as") and can be performed
automatically on the approx. 230-240 uses. I'm not sure how many of
these are already tested, though. Some other things also changed about
Exceptions, but these I will address separately.


That's as far as I got so far. As I said above, most of these are
simple syntax changes which are hard to get wrong. If we opt for the
2to3 route, all of those would be converted automatically but it is
still preferable to get the code in a more "modern" state. Of course,
if not using 2to3, all of the above changes will have to be made.
Waiving the tests requirement would allow me to progress much faster,
and, at the end of the day, my task is to add Python 3 support, not
improve test coverage to 100%.

Thank you all in advance for any input (or help!).




[1] If you are interested, the best way of following my progress is
probably to subscribe to my blog (which is also syndicated on Planted
Twisted):
vperic.blogspot.com

[2] The merits of each approach can be discussed elsewhere, this is
not the goal of this e-mail. As of right now, there is sufficient work
remaining that will be equally beneficial no matter which one is
chosen, and this is the stuff I'm focusing on now.

-- 
Vladimir Perić