[perl #60642] [CAGE] add a codingstd test to ensure TODOed tests have an RT ticket number

[James Keenan via RT]

On Wed Nov 19 23:13:27 2008, mark at glines.org wrote:
> James Keenan via RT wrote:
> > On Tue Nov 18 10:22:25 2008, mark at glines.org wrote:
> > 
> > This will probably be quite challenging.  Let's assume that all tests
> > are found in files with names ending in '.t'.  Those .t files can be
> > written in any language, which probably have different ways of
> > classifying a test as TODO.
> > 
> > My count tonight is that 1384 .t files in the distribution.  Of these
> > 524 are *not* found under ./languages/.
> > 
> > I wonder if we could formulate the specification in this ticket a bit
> > more precisely before someone embarks on coding.
> 
> Yes, absolutely.  I just added the basic ticket on the spur of the 
> moment, to make sure I didn't forget about it.
> 
> I've been thinking about this.  A few things come to mind, for instance 
> detecting the language based on the hashbang (if any) or subdirectory 
> it's in, and invoking a language-specific parser.  And detecting the 
> cases we can't handle, and skipping those.
> 
> But to me that sounds like way too much work.  It doesn't really matter 
> to me whether the ticket number occurs within the TODO output string, a 
> nearby comment is good enough for me.  So how about skipping all the 
> above nonsense and just ignoring the test language entirely?  How about 
> a simple regex-based test that tallies all instances of /TODO/ in the 
> set of test files, skipping the lines that start with obvious comment 
> characters, and for each instance, looks for a match of /#\d+/?  It can 
> even expand the search to also look a couple lines above and below the 
> TODO line, for additional flexibility.  I think that should be 
> reasonable for most, if not all, possible test languages.
> 
> Do you think that would catch all the cases?  


Two thoughts:

1.  We already have code that can detect the existence of TODO in
certain kinds of files.  Cf. t/codingstd/fixme.t.  Paul Cochrane used
that a couple of years back to generate hundreds of RTs -- most of which
are probably still outstanding.  Can we leverage that
Parrot::Distribution-based code?

2.  I've heard a lot of talk lately about languages moving into their
own repositories.  If so, then we have to ask whether we should be
instituting new coding standards for .t files under ./languages/.  At
what point do we say:  You (languages) are responsible for setting and
enforcing your own coding standards.  That would reduce the scope of
this ticket to files that are intrinsically related to Parrot.

Feedback welcome.  I'm not wedded to any approach and am not committing
myself to any.

kid51