<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><blockquote type="cite" class="">On Jan 12, 2015, at 12:49 PM, Tobias Oberstein <<a href="mailto:tobias.oberstein@tavendo.de" class="">tobias.oberstein@tavendo.de</a>> wrote:<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Probably the test machine was changed, a new version of OpenSSL or<br class=""></blockquote>pyOpenSSL, or something else?<br class=""><br class="">One of those things. There is no infrastructure in place for identifying events<br class="">which impact the performance testing infrastructure. The only performance<br class=""></blockquote><br class="">Yes, this is an important point: track changes in infrastructure (everything that might<br class="">have an influence, but is outside the tested code).<br class=""><br class=""><blockquote type="cite" class="">testing environment is a very old mac mini still running Snow Leopard, which<br class=""></blockquote><br class="">omg;)<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">I'd say: the infrastructure aspects when doing performance tests do<br class=""></blockquote>matter. To the degree that performance results are of very limited value at<br class="">all, if the former aspects are not accounted for.<br class=""><br class="">I don't think the results that we have presently are worth much at all. My<br class="">point was mostly that there is some infrastructure which is halfway usable,<br class="">and so you don't have to start from scratch. If you could take over this<br class=""></blockquote><br class="">You mean taking over the code "as is"<br class=""><br class=""><a href="http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files" class="">http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files</a><br class=""><br class="">or the task in general (Twisted benchmarking)?<br class=""></blockquote><br class="">Both, I think.<br class=""><br class="">I'd really prefer it if you could start with the existing benchmarks, incrementally maintain them (perhaps beginning by porting to Github, we seem to be having pretty good success with travis) and probably <i class="">eventually</i> replace them wholesale, than replace them from the beginning.<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">project (I am pretty sure at this point there is nobody to take it over *from*,<br class=""></blockquote><br class="">We are currently developing performance test infrastructure for Crossbar.io - naturally,<br class="">it is eating it's own dog food: the infrastructure is based on Crossbar.io and WAMP to<br class="">orchestrate and wire up things in a distributed test setup.<br class=""><br class="">We could extend that to test at the Twisted(-only) level. Need to think about how<br class="">that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't open-source.<br class=""><br class="">The testing hardware above (mac, no real network) is insufficient for what I need.<br class="">I'm thinking about buying and setting up 2 more boxes for Linux.<br class=""></blockquote><div class=""><br class=""></div><div class="">Keep in mind that a performance testing environment should be scalable. Others may have different environments they care about. Building your specific environment would be tremendously useful, but it would be even <i class="">more</i> useful to build it in a way that others can compare in their own hardware setups.</div><br class=""><blockquote type="cite" class="">Rgd. Codespeed (<a href="https://github.com/tobami/codespeed" class="">https://github.com/tobami/codespeed</a>), which seems to be used<br class="">by <a href="http://speedcenter.twistedmatrix.com" class="">speedcenter.twistedmatrix.com</a>: I have issues here as well.<br class=""><br class="">E.g. I need latency histograms, but this seems unsupported (benchmark results can<br class="">only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. <br class="">Throughput is one thing. Constistent low latency a completely different. The latter is<br class="">much much harder.<br class=""><br class="">But what is the "interface" between test cases from "twisted-benchmarks" to codespeed?<br class=""></blockquote><br class="">Codespeed runs the benchmark, and they print out this stuff: <a href="https://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head:/benchlib.py#L12" class="">https://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head:/benchlib.py#L12</a><div class=""><br class=""></div><div class="">POSTing them via JSON would be nicer, structured data is great.<br class=""><br class=""><blockquote type="cite" class="">This<br class=""><br class=""><a href="https://github.com/tobami/codespeed#saving-data" class="">https://github.com/tobami/codespeed#saving-data</a><br class=""><br class="">seems to suggest, performance test results are HTTP/POSTed as JSON to codespeed.<br class=""><br class="">And codespeed is then only responsible for visualization and web hosting, right?<br class=""><br class="">I think we can find something better for that part.<br class=""><br class=""><blockquote type="cite" class="">(And if you care a lot about performance in a particular environment you<br class="">could set it up in that environment and get attention for it :)).<br class=""></blockquote><br class="">Yes, in particular that very last one is a factor to justify efforts;) Anything like having<br class="">a promo logo or similar - that would be an argument to invest time and material.<br class="">I will seriously contemplate .. need to align with strategy/available time.<br class=""></blockquote><div class=""><br class=""></div><div class="">You should probably contact tsf@ with these concerns :).</div><br class=""><blockquote type="cite" class="">We already host FreeBSD buildslaves for both Twisted and PyPy. That might be another<br class="">synergy (hosting the latter on that same boxes).<br class=""><br class=""><blockquote type="cite" class="">You should also have a look at the existing benchmark suite and potentially<br class="">look at maintaining / expanding that as well.<br class=""></blockquote><br class="">I will try to integrate some of this into our upcoming perf. infrastructure.<br class=""><br class="">/Tobias<br class=""><br class=""><blockquote type="cite" class=""><br class="">Thoughts?<br class=""><br class="">-glyph<br class="">_______________________________________________<br class="">Twisted-Python mailing list<br class=""><a href="mailto:Twisted-Python@twistedmatrix.com" class="">Twisted-Python@twistedmatrix.com</a><br class="">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python<br class=""></blockquote><br class="">_______________________________________________<br class="">Twisted-Python mailing list<br class=""><a href="mailto:Twisted-Python@twistedmatrix.com" class="">Twisted-Python@twistedmatrix.com</a><br class="">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python<br class=""></blockquote><br class=""></div></body></html>