[RFC][PATCH] Rename Rakudo's "spectest_regression" to "spectest"

[Allison Randal]

Moritz Lenz wrote:
> jerry gay wrote:
> A combined harness is much better in terms of reporting.

Yes.

>> the tests we expect to pass reliably should be the default tests we
>> run. we expect all spectest_regression tests to pass reliably. the
>> default test target should always be named 'test'. it seems natural
>> that we add spectest_regression to the 'test' makefile target.
>> additionally, this would have possibly prevented the 74 failures
>> post-mdd-merge, since allison didn't know about the additional test
>> target in the makefile.
> 
> well, if reading the README is too much even for our architect then we
> shouldn't assume that anybody else does ;-)

Another thing that would be helpful for languages in trunk is something 
like a TESTME file. It should briefly say exactly what steps a core 
developer should take to test that their changes haven't broken the 
language, and if failures are expected or all tests should pass. Also, 
anything strange like having multiple test harnesses running in sequence 
instead of aggregating the results in one report. (That one caught me on 
Rakudo's 'make test' too. I thought all the test were passing, and then 
found that the final "All tests pass" report was hiding earlier failures 
in a different summary.) The README is quite verbose and intended for 
people who want to use the language. Even after reading it, it's not 
straightforward to decide what to test and whether failures are relevant.

What would be really ideal is if core developers could just run 'make 
languagetest' in the repository root and get a single report of all the 
language failures, and know for sure that any failures are their 
responsibility. But, we're a long way away from that.

Allison