[perl #60642] [CAGE] add a codingstd test to ensure TODOed tests have an RT ticket number

[Will Coleda via RT]

On Thu Nov 20 04:16:53 2008, jkeen at verizon.net wrote:
> On Wed Nov 19 23:13:27 2008, mark at glines.org wrote:
> > James Keenan via RT wrote:
> > > On Tue Nov 18 10:22:25 2008, mark at glines.org wrote:
> > > 
> > > This will probably be quite challenging.  Let's assume that all 
tests
> > > are found in files with names ending in '.t'.  Those .t files can 
be
> > > written in any language, which probably have different ways of
> > > classifying a test as TODO.
> > > 
> > > My count tonight is that 1384 .t files in the distribution.  Of 
these
> > > 524 are *not* found under ./languages/.
> > > 
> > > I wonder if we could formulate the specification in this ticket a 
bit
> > > more precisely before someone embarks on coding.
> > 
> > Yes, absolutely.  I just added the basic ticket on the spur of the 
> > moment, to make sure I didn't forget about it.
> > 
> > I've been thinking about this.  A few things come to mind, for 
instance 
> > detecting the language based on the hashbang (if any) or 
subdirectory 
> > it's in, and invoking a language-specific parser.  And detecting the 
> > cases we can't handle, and skipping those.
> > 
> > But to me that sounds like way too much work.  It doesn't really 
matter 
> > to me whether the ticket number occurs within the TODO output 
string, a 
> > nearby comment is good enough for me.  So how about skipping all the 
> > above nonsense and just ignoring the test language entirely?  How 
about 
> > a simple regex-based test that tallies all instances of /TODO/ in 
the 
> > set of test files, skipping the lines that start with obvious 
comment 
> > characters, and for each instance, looks for a match of /#\d+/?  It 
can 
> > even expand the search to also look a couple lines above and below 
the 
> > TODO line, for additional flexibility.  I think that should be 
> > reasonable for most, if not all, possible test languages.
> > 
> > Do you think that would catch all the cases?  
> 
> 
> Two thoughts:
> 
> 1.  We already have code that can detect the existence of TODO in
> certain kinds of files.  Cf. t/codingstd/fixme.t.  Paul Cochrane used
> that a couple of years back to generate hundreds of RTs -- most of 
which
> are probably still outstanding.  Can we leverage that
> Parrot::Distribution-based code?

If we're really going to enforce this, we should write a TAP parser that 
verifies that any TODO/SKIP messages refer to a ticket. But don't do 
that.

I think it's overkill to work on this, and would much rather see time 
spent un-TODO-ing and un-SKIP-ping tests. In fact, in the time it takes 
for us to figure out a programmatic way to do this, we could probably 
just do it by hand.

-1
-- 
Will "Coke" Coleda