Euler Project Problem #4 Benchmark on Parrot 0.2.0 - trunk (r42903)

[chromatic]

On Sunday 06 December 2009 at 08:06, Geoffrey Broadwell wrote:

> While I would love for HEAD to be always the fastest, if you mentally
> graph this you can see that trunk is in the main grouping of total times
> in the 32 to 40 second range.  It's the outliers in the 55 to 74 second
> range that really aren't looking good.
> 
> To me there are two optimization goals:
> 
> 1. Make sure HEAD is in the same grouping as the fastest time, for all
> benchmarks we can find.  (Don't ask me for an exact rule for grouping --
> but if you need a programmatic rule of thumb for a meta-bench tool,
> perhaps cut off at 30% or even 50% worse than the best time.)

I don't trust time-related benchmarkings for this reason.  Yes, stochastic 
analysis can give you some degree of probability that comparisons are all 
like-to-like, but the output of a single Callgrind run is much easier to 
compare (and gives a better explanation of *why* and *how* and *where* 
performance has changed).

-- c