the future of the build process

[Christoph Otto]

The pmc_pct branch is bacek and my effort to implement a pmc compiler in PCT. 
  This is step 1 in chromatic's plan for world domination and a self-hosting 
Parrot[1].

The branch is at the point when we need to start thinking about what the 
future build process will look like once Parrot is self-hosting.  There are 
two main possibilities I can see.  The first is that we check in the minimal 
subset of generated files needed to build a minimal Parrot which can bootstrap 
the full Parrot.  The second is that we check all generated files in to svn. 
Both have their trade-offs.

If we go with the minimal approach, I picture a typical build going something 
like this:
* build minimal parrot, possibly with out-of-date generated files from svn
* build nqp, pmcc and other Parrot-based build tools with the minimal Parrot
* generate all files (pmcs, ops, headers, etc)
* build full Parrot, now with up-to-date generated files
* (possibly) rebuild nqp, etc
This has the advantage that it will require fewer generated files to be 
checked into svn and makes it less likely that stale files will stay around. 
The downside is that it would mean a slower build process since Parrot would 
need to be built twice.  We'd also have to keep track of which PMCs are 
needed.  By my count 36 of the 78 current PMCs are currently used during the 
build [2].


If we go with the approach of checking in all generated files, it will mean a 
generally faster build, but changes that affect generated files will be a 
little more painful than they currently are.  The build process for changes 
that don't affect generated files will remain the same.  If generated files 
get touched, the process (from make realclean) will look something like:
* generate Parrot from out-of-date generated files
* build nqp, etc
* makefile dependencies notice that something needs to be regenerated
* regenerate files with Parrot-based tools
* compile Parrot again with regenerated files
* (possibly) rebuild nqp, etc.

I'm more inclined toward the second option, but that's just what I see from 
where I'm sitting.

In both cases I'd propose that all necessary checked-in generated files go 
someplace like src/gen/pmc, etc to minimize the size and frequency of changes 
involving generated files.  It would also make life easier if there were a 
make target to (almost) automatically commit changed generated files, since 
any changes to either the PMCs or the build tools would require such updates.

There's more to consider, especially how to minimize unnecessary rebuilds, but 
that can wait until we have a decision fleshed out on this.

Christoph




[1] http://irclog.perlgeek.de/parrot/2009-04-21#i_1083550

[2] I added code to include/parrot/vtable.h to spit text into a file whenever 
VTABLE_init and VTABLE_init_pmc were called.  The following PMCs were used at 
least once during the build process:
AddrRegistry, Array, Boolean, CallSignature, Capture, Class, CodeString, 
Coroutine, Eval, ExceptionHandler, File, FileHandle, FixedIntegerArray, 
FixedPMCArray, FixedStringArray, Hash, Integer, Iterator, Key, LexInfo, 
LexPad, ManagedStruct, MultiSub, NameSpace, NCI, OrderedHash, PMCProxy, 
ResizableIntegerArray, ResizablePMCArray, ResizableStringArray, 
RetContinuation, Scheduler, String, Sub, Undef, UnManagedStruct