[PROPOSAL] include nqp-rx PIR files in Parrot core distribution

Patrick R. Michaud pmichaud at pobox.com
Sat Oct 31 06:04:45 UTC 2009


Several people are asking about the possibility of making
the nqp-rx libraries and new NQP compiler available in the
core Parrot distribution.  Generally I think doing this will
make things much easier for HLL developers, but I can also
envision that we might want to treat it entirely as an
external library.

This message is intended to describe the overall architecture 
of the nqp-rx compiler tool suite, and to propose a way in
which these could be made available in core Parrot.

Before describing the architecture, a few key points:

* NQP-rx is intended to replace PGE, the existing NQP,
  and some portions of PCT.  However, in keeping with
  Parrot's deprecation policy, the existing PGE/NQP/PCT
  libraries will remain intact until at least 2.0 .

* I intend to eventually drop the "-rx" portion of the
  name; it exists now simply to make it easier to distinguish
  the new NQP from the old one.

* At this time I don't have any intent to merge the full 
  nqp-rx source code repository into the Parrot source repository.  
  Parrot may of course choose to incorporate nqp-rx sources via the
  Artistic License, but for the foreseeable I view NQP's mainline 
  development as being outside of the Parrot repo.

With those out of the way, here's the overall architecture
of the nqp-rx toolkit.  Basically there are four key components:

1.  Regex.pbc -- the core of the regex engine.  This is
    largely analogous to (and intends to ultimately replace)
    the existing PGE.pbc engine.

2.  HLL.pbc -- a core set of libraries for HLL compilers.
    This includes a base compiler class (HLL::Compiler) and
    a base grammar (HLL::Grammar) for language parsers,
    ultimately intended to replace PCT::HLLCompiler and
    PCT::Grammar.

3.  P6Regex.pbc -- a Perl 6 regular expression compiler.
    This compiles Perl 6 regular expressions into subroutines
    that (in conjunction with Regex.pbc above) can perform
    pattern matching.

4.  nqp.pbc -- a language for writing compilers and libraries
    that uses a limited Perl 6 syntax.  NQP statements that
    use regexes and/or grammars (e.g., parsers) requires
    the Regex.pbc library above at runtime, all other NQP
    code is translated directly to Parrot opcodes without
    needing additional runtime support.

The intent is that someone writing a HLL compiler will code
the parser, ast transformations, and perhaps builtin functions
entirely with NQP-rx; once the compiler is generated it needs
only the Regex.pbc, HLL.pbc, and PCT/PAST.pbc (from Parrot)
libraries for execution.  (If the compiler also needs runtime 
compilation of Perl 6 regular expressions, it will also 
want P6Regex.pbc.  Perl 5 regular expressions will be available
via P5Regex.pbc, soon to be written.)

Fundamentally this is _exactly_ the same approach we've been
using with PCT until now; the main difference is that
instead of having separate programs for building the
parser (PGE/Perl6Grammar.pbc) and the ast transformation (NQP),
it can now all be done with a single (and much improved) 
NQP-rx compiler.  

So, a HLL compiler writer can use NQP to create the compiler;
once built NQP itself is no longer needed.  But the Regex.pbc
and HLL.pbc files are still needed, since every HLL will tend
to need these files at runtime, it makes some sense to bundle
them as part of core parrot.

Fortunately, including all of these components as part of Parrot 
can be fairly straightforward.  Inside the nqp-rx repository, 
each of the four .pbc libraries above are compiled from a 
generated .pir file that has no dependencies beyond what 
already exists in Parrot.  

So, my proposal would be to periodically grab copies
of the generated .pir files from the nqp-rx repo into a 
compilers/nqp-rx/ directory in Parrot, and add commands to
Parrot's build subsystem to compile the .pirs to .pbcs and
install them in the appropriate directories.  (nqp.pbc would
likely be installed as nqp-rx.pbc for a time, to avoid conflicts
with the existing nqp.pbc.)

We can also include copies of nqp-rx's test suite into the Parrot
repository, so that NQP and Parrot developers can determine
that changes to NQP and/or Parrot aren't causing regressions
for end-users.

Parrot can control the timing of NQP updates into its
repository, consistent with Parrot's release and support
policies.  There's no need for it to keep up with NQP's
bleeding edge.  That said, based on our recent experience with
nqp-rx I don't foresee NQP as having a significant bleeding
edge or issues with backwards compatibility.  Earlier today 
Jonathan was able to switch Rakudo from using the old NQP 
to the new one with only very minor (and easily documented) 
changes to Rakudo's existing code base.  It turns out that
nqp-rx is surprisingly compatible with what went before.
And again, as mentioned above, I'm not proposing that we
immediately replace the existing PCT/PGE/NQP libraries,
but rather to live in parallel for a while to give HLL 
developers an opportunity to migrate to the new toolset
at whatever pace works best for them.

I will note that the new toolset is significantly improved
over what we have now, with more improvements to come
(especially performance).  So I expect HLL and library developers 
to migrate to the new toolset quickly.

If Parrot core chooses not to ship with the nqp base libraries,
then it essentially means that every installation will need
a mechanism to obtain the libraries from the nqp-rx repo or
some other source.  I grant that in the general case packagers 
can do the work of managing and bundling the needed library
.pbcs, but for HLL and library developers it's much more
straightforward if there's already versions of the files
directly available from the Parrot repo.  

If I hear general approval for this proposal, I'll quickly
incorporate nqp-rx into the Parrot repository (this is
fairly straightforward and not at all complicated).  If
there's general rejection of the proposal, then we'll
start building tools for independently downloading, installing,
and maintaining appropriate NQP versions in a parrot installation
(this is quite a bit more complex, which is why I'm writing 
this in hopes of avoiding the complexity).

Comments and questions welcomed, and thanks!

Pm


More information about the parrot-dev mailing list