A plea from Rakudo-land

[Patrick R. Michaud]

As I've mentioned in the last couple of parrotsketch meetings, over
in Rakudo-land we're really seeing a large increase in memory-related
errors in recent Parrots.  Today's daily spectest run recorded 90
test failures:

  spectest-progress.csv update: 412 files, 11546 passing, 90 failing
  Failure summary:
    S05-grammar/parse_and_parsefile.t aborted 6 test(s)
    S05-metasyntax/regex.rakudo aborted 17 test(s)
    S12-enums/basic.rakudo 27 - short name of the enum without ...
    S16-filehandles/io.rakudo aborted 62 test(s)
    S32-num/rand.rakudo aborted 4 test(s)

The S12-enums/basic.rakudo and S32-num/rand.rakudo files are true
Rakudo bugs, and are already being addressed.  But the other 85 failing
tests represent places where this test run encountered problems that
are seemingly due to memory corruption issues in Parrot.

I'll provide more details in just a bit, but my overriding question
is:  How can we report these problems in a way that make it easier for
others to diagnose (and thus more likely to be fixed)?  The standard
answer ("provide a short PIR snippet") doesn't seem to be viable here;
in most cases I find that simply removing a line of Perl 6 source
or adding a comment is sufficient to cause the underlying error to
disappear altogether.  And, of course, any specific failure seems
be repeatable only for a specific revision of Parrot and Rakudo,
such that testing against any other configuration doesn't expose
the problem.

While waiting for an answer to the above question, I'll use
the rest of this message to describe (in gory detail) the current
set of failures I'm seeing.  For the results below I'm starting
from completely fresh checkouts and builds of Rakudo and Parrot,
built as follows:

  $ git clone git at github.com:rakudo/rakudo.git
  $ cd rakudo
  $ perl Configure.pl --gen-parrot
  $ make 
  $ make spectest

Rakudo is git revision 6a4d66, Parrot is svn revision r39868.
I'm performing the builds on x86 and x86_64 versions of Kubuntu 9.04.
Both the 32-bit and 64-bit versions result in spectest failures, but
fail in different places.

For any specific test there are at least four different ways we can 
run the test -- depending on the way it's invoked we can get different
results:

  $ make t/spec/S05-metasyntax/regex.t
  $ ./perl6 t/spec/S05-metasyntax/regex.rakudo
  $ parrot/parrot    perl6.pbc t/spec/S05-metasyntax/regex.rakudo
  $ parrot/parrot -G perl6.pbc t/spec/S05-metasyntax/regex.rakudo

When invoked using this final form (with the -G flag to Parrot),
all test failures disappear entirely, regardless of platform.

For example, here are the results of the above four commands
on the S05-metasyntax/regex.t test above:

                  x86_64    x86
   make            abort   pass
   perl6           abort   pass
   parrot          abort   pass
   parrot -G       pass    pass

Here are the four commands on the S05-grammar/parse_and_parsefile.t 
test.  Note that unlike the above, this test passes without the -G 
flag to parrot, but still fails in the perl6 fakecutable:

                  x86_64    x86
   make            abort   pass
   perl6           abort   pass
   parrot          pass    pass
   parrot -G       pass    pass

Lest we think this is just a 64-bit problem, here's S04-statements/do.t

                  x86_64    x86
   make            pass    abort
   perl6           pass    abort
   parrot          pass    abort
   parrot -G       pass    pass

A bizarre note about S04-statements/do.t -- it's only the fudged version
(do.rakudo) that aborts.  The unfudged version (do.t) runs to completion 
on x86 without aborting, albeit with the (expected) 10 test failures.

Here are the full spectest results in the x86 and x86_64 environments

x86:
Failed Test                               Stat Wstat Total Fail  List of Failed
-------------------------------------------------------------------------------
t/spec/S02-names_and_variables/names.raku               ??   ??  ??
t/spec/S03-operators/p5arrow.t                          ??   ??  ??
t/spec/S03-operators/smartmatch.rakudo                  ??   ??  ??
t/spec/S04-statements/do.rakudo                         ??   ??  ??
t/spec/S06-signature/named-parameters.rak               ??   ??  ??
t/spec/S10-packages/basic.rakudo             0     6    22    0  ??
t/spec/S12-enums/basic.rakudo                           30    1  27
t/spec/S32-array/exists.rakudo                          ??   ??  ??
2142 subtests skipped.
Failed 8/412 test scripts. 1/14009 subtests failed.

x86_64:
Failed Test                               Stat Wstat Total Fail  List of Failed
-------------------------------------------------------------------------------
t/spec/S02-literals/array-interpolation.r    0    11    12   12  7-12
t/spec/S05-grammar/parse_and_parsefile.t                ??   ??  ??
t/spec/S05-metasyntax/charset.rakudo                    ??   ??  ??
t/spec/S05-metasyntax/regex.rakudo                      ??   ??  ??
t/spec/S10-packages/basic.rakudo             0     6    22    0  ??
t/spec/S12-construction/autopairs.t          0     6     4    2  4
t/spec/S12-enums/basic.rakudo                           30    1  27
t/spec/S32-array/push.rakudo                 0     6    44    0  ??
t/spec/S32-list/map.rakudo                   0    11    89  150  15-89
t/spec/S32-num/rand.rakudo                   0     6   110    0  ??
t/spec/integration/99problems-31-to-40.ra    0    11    67    4  66-67
2208 subtests skipped.
Failed 11/412 test scripts. 85/14199 subtests failed.

Personally, I really hope we can get this locked down before the 1.4
release.  Either that or I suspect we'll want to get the tests
(and perhaps Rakudo itself) to primarily run with the equivalent of
Parrot's -G flag in place.

Thanks,

Pm