Hilarious dream-logic (was Re: [Twisted-Python] [patch] (etc)

[Andrea Arcangeli]

On Fri, May 19, 2006 at 10:41:35AM +1000, Justin Warren wrote:
> break existing code, ie: regression testing. I'm personally a big fan of
> this feature of test suites.

I'm a big fun of test suites as well. I only disagree with wasting time
delaying the integration of a valid bugfix just because the unit test
doesn't exist yet. I absolutely never said unit tests are a waste of
time. For the last months I've always said that "unit tests" are
_welcome_ at the top of the cpushare-twisted webpage.

> Incidentally, those with slightly more CS knowledge know that it is
> possible, though by no means easy, to build a system that is provably
> correct. An investigation of the Z specification language may prove
> enlightening. I think you mean that a test suite proves that the system
> passes all the tests, not that it is bug free.

What I mean is that a test suite cannot prove the code is bug free. Nor
that any bugfix is correct. If nothing else because the test suite may
be buggy too. This is obvious.

Clearly a test suite is welcome and can only help, but its mandatory
requirement for any change to the code sounds way excessive.

I perfectly know about formal demonstrations being possible too (I spoke
about those matters for a long time last year in a completely different
context) but they're not unit-tests (certainly not the ones you see in
the twisted reposistory), so I didn't mention this to avoid further
confusion. I doubt it's feasible to demonstrate Twisted bug free
formally (to back my guess I remind you Alan Cox quote saying twisted is
a 6m unauditable weirdness, I guess he was partly joking though).

	http://article.gmane.org/gmane.linux.kernel/327172

My current worries are the troubles with poll, I worry about the lack of
epoll, I worry about scaling in SMP with one thread per cpu. Those are
the things that should be discussed instead of receiving emails from
people about lack of unit tests for fixes that can be trivially verified
by reading the code.

> make it so. If you have benchmarks that demonstrate your claim, and
> importantly, the method used to generate them, then by all means share
> them with others. If your claims are true, this information will help in
> fixing the problem. Fixing these bugs is what we all want, after all.

About the benchmarks to make an example I posted some benchmarks here:

	http://twistedmatrix.com/pipermail/twisted-web/2006-January/002425.html

I used the klive homepage for it. You can reproduce yourself downloading
it (all GPL):

	http://klive.cpushare.com/downloads/

Older versions uses web+nevow, newer uses web2+Cheetah (I don't remember
exactly the time of the switch but it's easy to find with some diff).

However here the kind of the answers I got:

	http://twistedmatrix.com/pipermail/twisted-web/2006-January/002428.html

So I didn't post more benchmarks, nor I tried to produce an official
benchmark that is easier to run than to install klive locally. Also
note, quite a lot of the time of klive is spent in the database. So it's
one of the worst possible benchmarks for web1+nevow vs web2+cheetah
since only little time is spent for the rendering. I measured much
higher html delivery speedups in other pages that weren't asking the db
such cpu intensive queries.

If there is interest I can produce an official benchmark (that sounds
much more useful than unit-tests for every bugfix). However I guess
we'll have to follow the above advice and do it on the cpushare-twisted
list. Also note, if you've a better place than cpushare-twisted for
things that may not be welcome here, that's fine with me. I made up this
cpushare-twisted things to fit my needs, only because I didn't find any
other alternate place.