Rakudo notes for 2011.05 Parrot Developer Summit

Tue May 17 06:12:02 UTC 2011

[Note:  This message was provided to Parrot Developer's Summit
participants near the start of Saturday's summit meeting; I'm
reposting it now to parrot-dev so we'll have a more permanent
archive of the message than the gist nopaste that was used.
Comments and feedback on this article are of course welcome.  --Pm]

As a result of the last PDS in January, I sent out an email identifying
what Rakudo needs from Parrot in 2011 [1].  This is an interim update
to give my views on where Parrot has improved or not improved for
Rakudo since that posting.

The posting identified five things that Rakudo needed from Parrot:
0) anything that makes Parrot/Rakudo faster overall
1) improved garbage collection
2) fixes to profiling tools and documentation
3) serialization
4) create .pbc directly from a Parrot program

Since that posting, excellent progress has been made on garbage 
collection (#1 above) -- commendations to all involved,  especially 
bacek++.  However, recent tests and benchmarks have shown that the 
primary motivation for the better GC -- that is, increased speed (#0) --
hasn't materialized on Parrot yet.  I discovered this when creating
the 2011.04 Rakudo Star release -- based on messages to the mailing
lists I was expecting around a 20% performance improvement in Rakudo 
and Parrot; instead I found a slight degradation in performance on nearly
all of the machines I tested.  Since Rakudo has been relatively stable
over this period of time, most (but certainly not all) of the variation
in performance would seem to be attributable to changes in underlying
Parrot performance.

As a result of the above I've created a Rakudo benchmarking platform [2] 
that makes it easier to evaluate different Rakudo/Parrot configurations
over time.  The latest results (from this morning) can be seen at

https://github.com/pmichaud/rakbench/blob/master/results/kiwi-x86_64-201105140653.txt
https://github.com/pmichaud/rakbench/blob/master/results/plum-x86_64-201105140651.txt
https://github.com/pmichaud/rakbench/blob/master/results/orange-x86_64-201105140653.txt

What these results seem to show is that the new GC only helps in those
places where GC was particularly problematic to begin with, such as
the regex tests.  In most other cases, released Parrots since January
have performed worse (and sometimes *much* worse) than the version released 
in January of this year.  Indeed, it was these bencmarks that pointed
out some problems that ultimately led to the fixes that show the
performance gains for rakudo-master and rakudo-bleed on some platforms.

Bottom line:  In spite of the new GC, overall performance has not 
really improved since January.

The new rakbench suite underscores just how important it is that Parrot
get some reliable profiling tools in place (#2).  The ones that exist now
aren't at all reliable, as detailed in my January message.  Yesterday 
I inquired on #parrot about the state of profiling, and was told that 
since there were no GSOC takers for this task that Parrot will have to 
formulate a plan to work on them.  From Rakudo's perspective, this 
continues to be an immediate and dire need, so I hope Parrot will do so.

Yesterday evening I hacked together some very Rakudo-specific 
quick-and-dirty PIR-level-sub call count capabilities, and was 
able to almost instantly identify a 10% overall performance 
improvement for Rakudo by replacing a frequently-called 1-line 
method with a much faster 3-line version.  Even better improvements
should be forthcoming with more investigation.  Given that there 
seems to be little movement on profiling on Parrot, we'll continue 
to develop the Rakudo-specific version and also likely instrumenting 
the new NQP with similar capabilities.  But Parrot ought to be able 
to do a far richer form of sub-level profiling than what we can do,
and when it's available we'll immediately be able to take advantage
of it.

Creating .pbc directly from a Parrot program (#4) is being worked on
satisfactorily, and although I haven't seen any direct effort towards
serialization (#3), I tend to view that a longer-term goal rather than
something that is likely to happen immediately.

Indeed, it should be restated that all of the items in my January
message were being presented with "we'll be glad to see them
whenever they can be made available" and no real expectations of
delivery dates.  That continues to the be the case-- anything Parrot
can do to improve the items listed above will be help Rakudo
immensely.  It also continues to be the case that overall speed
is critically important for us, and that thus far Parrot releases
have trended in the wrong direction on that front.

[Post-PDS note:  The items described below were discussed extensively
during PDS, and have led to new policies to establish 'relationship
manager' roles to manage them.  See [3] for the followup.]

Lastly, there continues to be a great deal of friction between the
Parrot and Rakudo teams regarding stability and deprecation.  I don't
have any direct answers.  I do know that I personally (and several
of the other Rakudo team members)  are weary of all of the revisiting
(I call it "re-litigating") of design, implementation, and process 
decisions that were supposedly "settled" by discussions long ago  
and even not-so-long ago.  I can provide a list of detailed and 
recent examples of such "revisits" if we wish to analyze this further.  

As I've repeatedly stated in the past -- Rakudo developers know that 
breakages will occur, and that's overall okay with us.  What hurts 
is that when we report the breakages, either after they occur or
because we note that a proposed change to Parrot will cause us
a serious issue, we end up spending hours and hours of collective
discussion time trying to _justify_ to a few Parrot developers why
the breakage is painful and unwarranted before we actually get
someone in Parrot with sufficient authority/skill to relieve our
pain.  We don't mind the existence of breakages, we mind the 
absolutely herculean efforts we have to go through with various
Parrot developers to get them to stop denying there's an issue
(or that they've already addressed it) so that a mutual solution 
can be found.  Anything that can be done to alleviate this ongoing 
problem would be a big help.

[Post-PDS note:  See [3] to see what was decided on this issue.]

Pm

[1] http://lists.parrot.org/pipermail/parrot-dev/2011-January/005410.html
[2] http://github.com/pmichaud/rakbench
[3] http://lists.parrot.org/pipermail/parrot-dev/2011-May/005891.html