A new NQP roadmap, 2011-01-31

[Patrick R. Michaud]

Over the next couple of weeks we will be rolling out a new version of
NQP that is based on new backend code for object management and
multidispatch.  This document describes the changes we've identified 
for the new version of NQP, the expected impact on the existing 
Parrot/Rakudo/NQP ecosystems, and our expected timelines for the 
changes taking place.

Most importantly, the new NQP contains a shift in philosophy from
previous versions.  Previous versions (including "nqp-rx") aimed to
avoid any runtime component whatsoever, providing simply a HLL
interface to whatever capabilities were provided by the underlying
VM (i.e., Parrot).  The new NQP now explicitly expects some small
amounts of "library code" to be loaded on top of the VM, used to provide
a basic framework for object management, multidispatch, and regexes.
However, the overall goal of NQP continues to be a very minimal Perl 6
syntax for writing code libraries and compilers for VM backends
such as Parrot.

And yes, the new NQP is explicitly designed to enable other
languages implemented with NQP to run on multiple VM backends such
as the CLR, JVM, etc.  Several key people are already making important
contributions in this area now, and we expect more in the months
ahead.  But for the next few months our focus will be more on improving
Rakudo's performance and faithfulness to the Perl 6 specification
over trying to make it run on multiple platforms.  (Of course, if we
quickly discover that other VM backends are likely to give significant
performance benefits sooner, we may increase our efforts in this area.)

To simplify the transition, the new NQP will be hosted in a new
GitHub repository separate from the existing perl6/nqp-rx repository.
This should enable the old nqp-rx line to be maintained and available
for those systems that depend on it, while allowing the radical
changes to take place in the new one.  In order to (hopefully)
avoid confusion, from now on we'll consistently use "nqp-rx" to
refer to the existing implementation, and "nqp" to refer to the
new one.

The new nqp implementation provides its own object metamodel
implementation (commonly known as "6model"), avoiding almost entirely
the object framework currently provided by Parrot.  In other
words, the new nqp does not make use of Parrot's Class and Object PMC
types, nor does it use the P6object interface library.  Both the
Rakudo and Parrot development teams have identified in the past
that Parrot's object system needs some significant work; it's our
hope that nqp's implementation will be able to jumpstart this
for Parrot, perhaps by adopting it into its core in some manner.
The OO framework in nqp has been explicitly designed to be flexible
enough that other languages can use it to implement their metamodel
as well.  It makes very few assumptions at its core, and tries to
create a level playing field in terms of access to performance-improving
features needed by languages other than Perl 6.

The new nqp also contains a replacement implementation of multiple
dispatch -- it no longer makes use of the Parrot MultiSub PMC type.
The new dispatcher is more efficient and truer to the Perl 6
semantics (and hopefully flexible enough to easily support or
implement dispatch semantics for other languages as well).

As a result of these changes, nqp will also be embarking on
updated implementations of significant portions of the Parrot
Compiler Toolkit (PCT), to take advantage of the optimization
possibilities inherent in the new object and multidispatch
implementations.  In fact, we expect to (re-)implement PCT as nqp
source code in the nqp repository, borrowing liberally from
bacek's pioneering efforts in this area.  The new PCT will also
likely obtain a new name (yet to be determined), to avoid confusion
with the existing "Parrot Compiler Toolkit".  Existing references
to Q:PIR, pir::opcode(), :inline('...') and the like will be
phased out in favor of nqp::* generic equivalents.

Concurrently with the above changes, we will be migrating Rakudo
to use the new nqp as its underlying implementation platform.
This will be done as a branch of the Rakudo repository, much like
the "alpha->ng" migration that took place in late 2009/early 2010.
This migration is expected to be completed in the next 2-3 months.
In the meantime, the Rakudo "master" branch will continue to target
the existing nqp-rx and Parrot systems.  The next Rakudo Star release
is not scheduled until April 2011 -- as the new Rakudo branch nears
completion we will determine how it will impact the timing and content
of the next Rakudo Star release.

We expect that HLL translators and programs that used nqp-rx
primarily as a high-level implementation language will be able to
migrate to the new nqp with only minimal changes (and hopefully
substantial performance improvements).  Grammars shouldn't
require any significant re-work, and much of the action methods
and PAST construction should remain the same as before.  But the
devil for this will of course end up in the details, which we'll
deal with as we encounter them.

Parrot libraries and subsystems that are using nqp-rx to access
specific Parrot features will likely have more difficulty migrating
to the new nqp, as nqp abstracts away from the low-level Parrot
details a bit further.  Our plan recognizes and fully understands that
Parrot may elect to neither provide nor support nqp directly in its
distributions, and may even migrate existing tools and libraries
completely away from the existing nqp-rx and PCT.  Or, Parrot
might decide to embrace NQP more fully to take advantage of its
new optimization and compiler toolkit capabilities.  We can likely
find ways to preserve the ability for NQP to access low-level Parrot features
for Parrot-specific libraries.  (This becomes far more conceivable if
Parrot adopts the new object metamodel, which seems likely at this point
and is the major impetus behind the changes to nqp.)  Or, the existing
nqp-rx could continue to live on as its own project targeting Parrot
subsystems, perhaps even being adopted into the Parrot repository.
Whatever the Parrot team chooses to do in this area, nqp will
support as best it can within the goals and plans described above.

In particular, there is no expectation or request that the new
nqp will have to be bundled with Parrot in order to support Rakudo.
The new Rakudo branch will either pull or bundle its desired version
of nqp directly from the nqp repository, analogous to the way that
the current Rakudo master branch pulls from the Parrot repository
as specified by PARROT_REVISION.

We recognize this new NQP implementation will results in significant
changes to the existing Rakudo/Parrot/NQP ecosystems, but we're
expecting that the short-term pains will be far outweighed by the
performance and capability improvements the new design will offer.
We also know that the new implementations offer plenty of new
possibilities for -Ofun for many of us, and we hope that many other
people will want to join in the -Ofun as we explore the new
frontiers this makes available.

Feel free to pose questions and suggestions about NQP or this
plan to any of perl6-compiler, parrot-dev, #perl6, or #parrot.

Pm