Need simple benchmark for inter-branch comparisons of speed and memory

James E Keenan jkeen at verizon.net
Sat Nov 20 00:19:19 UTC 2010


As is well known, changes in Parrot during the last half-year have had 
the effect of increasing the amount of memory required to build Parrot 
in a reasonable amount of time.  This is particularly evident on 
machines that several years ago were perfectly adequate for building 
Parrot but are now considered short on resources (such as my vintage 
2004 iBook G4 with 256M physical memory).  With patches prepared by 
bacek, mikehh and others, I can get Parrot to build and run 'make test' 
on my small resource machine, but it does take a long time.

What I would like to have is a benchmark test I could run which, on the 
one hand, gives a small resource machine a good workout, but, on the 
other hand, is small enough for me to run it consecutively enough times 
to give me a statistically significant result in testing the various 
garbage-collection-focused branches.  For example, I know that 
regardless of which machine I test on, 
t/compilers/opsc/02-parse-all-ops.t takes longer than just about any 
file in our test suite.  If I were able to run it 30 times on master and 
30 times in a GC-branch, I would have a good measure of the degree to 
which a GC-branch improves over master (if indeed it does improve over 
master ;-) ).

The problem is that 02-parse-all-ops.t takes so long to run on 
small-resource machines that I can not, in practical terms, afford to 
run it 30 times in a row.  What I'd like is a test that takes say, about 
4 seconds to run on a small resource box (and, of course, much faster on 
most contemporary machines) so that I could run it 30 times in two 
minutes on a given branch.  That way, I could relatively quickly compare 
various branches.

Does that make sense?  Do we have such a test?

Thank you very much.

kid51



More information about the parrot-dev mailing list