[parrot-tickets] [Parrot] #357: Enable meaningful testing of t/native_pbc/*.t

[Reini Urban]

2009/3/9 Andy Dougherty <doughera at lafayette.edu>:
> On Mon, 9 Mar 2009, Reini Urban wrote:
>
>> > Changes (by doughera):
>> >
>> >  * status:  closed => reopened
>> >  * resolution:  fixed =>
>> >
>> >
>> > Comment:
>> >
>> >  I have reopened this ticket because you have not addressed the main
>> >  problem:  The tests done during normal development testing are not the
>> >  same as those done on a release.  To be specific:  Step 2g of the release
>> >  manager's guide instructs the release manager to change all the *.pbc
>> >  files.  Thus the files in the release will not be the ones that were
>> >  tested.
>>
>> Sorry, the main problem is already addressed.
>> It is in the guide, that the release manager should test and fix the
>> failing tests if required.
>>
>>   Please check with C<prove t/native_pbc/*.t>
>>
>> This was not done in 0.9.1
>>
>> Do you have better suggestions?
>
> Yes.  Redesign the tests so that step 2g is not required.  My fundamental
> objection is that if the instructions are followed, the files in the
> release will be *different* from those that have been tested prior to
> release.  Yes, the release manager can still test them on his or her
> system, but that's not the same as having them tested on a variety of
> systems in the weeks prior to release.  This is my main point.  The files
> in the release will not be the ones that were tested in the weeks prior to
> the release.
>
>> Not instructing to stamp the pbc's will not change anything, since the
>> tests only check the bytecode version, not the parrot version, so the
>> test logic remains the same.
>
> Running the update does *something*.  Try it.  You will find that after
> you run
>
>    perl tools/dev/pbc_header.pl --upd t/native_pbc/*.pbc
>
> tests that formerly passed now fail.  I don't know what is being changed
> or why they fail.

Oops. Indeed a new problem. Maybe because the uuid patch for packfile.c
is not in yet. I removed the bytecode version and uuid stamping from
pbc_header.pl as it is wrong anyway. I also fixed UUID but disabled it for now.
Done with r37243.

> I do know that such failures won't be discovered until
> step 2g of the release process, which may well be too late to do much
> about them or to adequately test any fixes.

I'm also in favor of this change. Though the changes to the pbc are
minimal, only the parrot version major, minor, patch) is updated, it
was not tested on the required different platforms and we cannot
burden a release manager with these coordination efforts.

{{{
--- docs/project/release_manager_guide.pod      (revision 37217)
+++ docs/project/release_manager_guide.pod      (working copy)
@@ -107,17 +107,17 @@
 =item g

 Coordinate 4-5 platforms to run C<bash tools/dev/mk_native_pbc>
-to update the native tests. Esp. when the PBC freeze state changed,
-when the tests fail. This happens quite frequently.
+to update the native tests early enough to get a good platform coverage,
+esp. when F<PBC_COMPAT> was updated.
 You'd need 32-bit and 64-bit, little-endian and big-endian platforms.
 linux-gcc x86_64 or solaris, plus sparc or darwin/ppc are usually enough.
 C<svn commit> the changed F<t/native_pbc/*.pbc> files.

-If not possible, run at least
-C<perl tools/dev/pbc_header.pl --upd t/native_pbc/*.pbc>
-to update version and fingerprint in the native tests.
-This is needed to mark pbc-incompatible changes.
-
+There's a tool C<perl tools/dev/pbc_header.pl --upd t/native_pbc/*.pbc>
+to update version and fingerprint in the native tests, but due to
+incomplete coverage of these changes on foreign platforms it is risky
+and should be avoided. We care only for the F<PBC_COMPAT> version and
+not for the parrot release version in these tests.
 Please check with C<prove t/native_pbc/*.t>.

 =item h
}}}




-- 
Reini Urban
http://phpwiki.org/              http://murbreak.at/