make fulltest_all

[Will Coleda]

On Mon, Apr 20, 2009 at 7:42 PM, François Perrad
<francois.perrad at gadz.org> wrote:
> 2009/4/21 James E Keenan <jkeen at verizon.net>:
>> Will Coleda wrote:
>>
>>>
>>> Please remember to either use 'make fulltest' or, if running 'make
>>> fulltest_all', to actually go back through and check the output, since
>>> fulltest_all actually ignores return values from the various
>>> harnesses, and theoretically always exits with a "success" return
>>> value.
>>>
>>
>> Coke:
>>
>> What then are we to make of this 'make fulltest_all' output I got this
>> morning:  http://nopaste.snit.ch/16326
>>
>> It clearly recorded two FAILs for harnesses.
>>

The shell exit value of 'make fulltest_all' is always true:

$ cat Makefile
notfalse:
        - at false

nottrue:
        - at true

false:
        false

true:
        true
$ make true; echo $?
true
0
$ make false; echo $?
false
make: *** [false] Error 1
2
$ make nottrue; echo $?
0
$ make notfalse; echo $?
make: [notfalse] Error 1 (ignored)
0

So, if all you're checking is the exit value, you'll miss the
failures. This is especially easy to do if the last several components
that are run pass, or if you're doing automated testing.

Like I said, you have to actually look at the output.

>
> t/examples/streams.t fixed by r38235.
>
> François

Thanks.

>> Or are we talking about different things?
>>
>> kid51

-- 
Will "Coke" Coleda