[parrot-tickets] [Parrot] #357: Enable meaningful testing of t/native_pbc/*.t

[Andy Dougherty]

On Mon, 9 Mar 2009, Reini Urban wrote:

> > Changes (by doughera):
> >
> >  * status:  closed => reopened
> >  * resolution:  fixed =>
> >
> >
> > Comment:
> >
> >  I have reopened this ticket because you have not addressed the main
> >  problem:  The tests done during normal development testing are not the
> >  same as those done on a release.  To be specific:  Step 2g of the release
> >  manager's guide instructs the release manager to change all the *.pbc
> >  files.  Thus the files in the release will not be the ones that were
> >  tested.
> 
> Sorry, the main problem is already addressed.
> It is in the guide, that the release manager should test and fix the
> failing tests if required.
> 
>   Please check with C<prove t/native_pbc/*.t>
> 
> This was not done in 0.9.1
> 
> Do you have better suggestions?

Yes.  Redesign the tests so that step 2g is not required.  My fundamental 
objection is that if the instructions are followed, the files in the 
release will be *different* from those that have been tested prior to 
release.  Yes, the release manager can still test them on his or her 
system, but that's not the same as having them tested on a variety of 
systems in the weeks prior to release.  This is my main point.  The files 
in the release will not be the ones that were tested in the weeks prior to 
the release.

> Not instructing to stamp the pbc's will not change anything, since the
> tests only check the bytecode version, not the parrot version, so the
> test logic remains the same.

Running the update does *something*.  Try it.  You will find that after 
you run

    perl tools/dev/pbc_header.pl --upd t/native_pbc/*.pbc

tests that formerly passed now fail.  I don't know what is being changed 
or why they fail.  I do know that such failures won't be discovered until 
step 2g of the release process, which may well be too late to do much 
about them or to adequately test any fixes.

-- 
    Andy Dougherty		doughera at lafayette.edu