[DRAFT] Parrot module ecosystem

[Allison Randal]

Geoffrey Broadwell wrote:
> 
> This is a VERY ROUGH OUTLINE of my proposal for the design of the Parrot
> module ecosystem.  I'm looking for any and all comments, and I expect
> many changes before we reach even rough consensus.  Please feel free to
> circulate this to people who understand the problem space better than
> I do!  I can use all the input I can get.
> 
> Note that there are some items explicitly marked 'undecided'.  I thought
> about these enough to realize I just don't know enough yet to make a
> decent recommendation, so input on these is especially appreciated.

A great proposal, thanks for taking the time to think through the issues!

> ************************************************************************
> VERSION: 2
> 
> Requirements
> ------------
> 
> * General
>   * Distributions should be dead easy for module authors to create,
>     and for users to install.
>   * We can create a centralized metadata store, but do not want to
>     build and manage a module distribution network ...

Good.

>   * However it should be possible for another group to do so.

<shrug> I'm not sure it is possible. Keep in mind that we're not just 
talking about custom Parrot modules, we're talking about the entire 
collected history of Perl, Python, PHP, Ruby, Lua, etc modules. That's 
just an insane amount of data, even Google isn't crazy enough to want to 
take that on.


And, an additional requirement:
  * The API for extracting info from the metadata store, and the process 
of installing modules based on that metadata should be dead simple and 
clearly specified so people can clone it. If we see implementations 
popping up in Python, PHP, Ruby, Perl (5&6), etc it'll be a mark of success.


> * Toolchain
>   * Basic tools can assume Parrot and the core modules are working,
>     but require no other dependencies internally.
>   * All external tools needed to download/build/install modules will
>     be specified in the module metadata.
>   * Tools should be easy to configure.
>   * Tools should attempt to auto-configure as much as possible.

Good.

>   * Tools must properly handle the difference between user-local,
>     site-local, and vendor-installed modules.

Not sure this really matters.

> * Metadata
>   * Simple, extensible format.
>   * Unicode and case-retaining.
>   * Must include its own spec version.

Good.

>   * Sufficient for automated programs to create system packages
>     (DEB, RPM, etc.).

That would be tough, but we can at least cover the simple cases, with 
guidance to point people toward how to extend the generated template to 
handle the harder cases.

>   * Separate static v. configure-discovered v. hand-edited metadata.
>     Separate files?

Sounds overly complex. Provide a field in the metadata for "data_source".

>   * Includes fetch, configure, build, test, install, and runtime
>     dependencies.
>   * Should be able to track author, mailing list, bug email/bug URI,
>     wiki, homepage, source repository, etc.

Good.

>   * Allows disambiguation as per Perl 6 module spec (authorities,
>     versions, authors, etc.).

It's just extra metadata, sure, why not. We should also make sure we can 
accommodate the metadata for RubyGems, PHP PEAR, etc.

>   * Specifies rules for dependency string parsing/interpretation.

Not sure what you mean. More detail?

> Proposal
> --------
> 
> * Overview
>   * Parrot community builds a module metadata search system.
>   * This search system gathers metadata from various sources, and
>     allows users to query it via web browser or API, but does not
>     itself store the actual modules.
>   * Once found, modules can be fetched from many possible sources,
>     including VCS repositories, FTP mirrors, etc.
>   * Parrot team will need to standardize module metadata, provide
>     the libraries and tools necessary to use the search system,
>     provide guidelines for extending the toolchain, and mentor the
>     growth of the ecosystem until it stands on its own.

Good.

> * Metadata format
>   * Served metadata container is gzip'ed tarball (.tar.gz? .tgz?).
>   * Core metadata is in META.json at top level of container.
>   * Container includes copies of special files (e.g. README).
>   * Format for specifying non-metadata-only build scripts undecided.
>   * Integrity check / authentication methods undecided.
>     * Probably at least md5sum and sha1sum for source tarballs,
>       but what about when pulling from raw VCS repo?

You're still thinking CPAN (the best technology 1995 had to offer).

The primary interface should be a web form where people can enter 
metadata about their module. They should also be able to *update* the 
information stored there, to mark an older version as deprecated, that a 
module is no longer maintained, change the owner(s), change the URI for 
download, or to remove a module entirely. (Look at Launchpad.net for 
inspiration.)

 From the form, we can generate a JSON dump of any module's metadata. We 
can also accept a JSON block as an alternate input source, so someone 
can keep a copy of the .json file checked into their repository, make a 
few changes and paste it in the web form when releasing the next version.

The metadata shouldn't keep copies of any files from the module 
distribution, though it should have space for a description. (If 
someone's lazy they might paste in the entire README, but that's 
generally about building a module, and so not appropriate for someone 
who's looking for general information about it, a.k.a. "Do I want this 
module?".)

> * Core modules
>   * parrot config  (already exists -- config.pir)
>   * HTTP client    (at least GET, with redirect and proxy support)
>   * zlib           (at least decompress)
>   * tar            (at least extract)
>   * JSON           (at least parse)
>   * version spec   (at least parse and compare)
>   * library probe  (shared library info: present? version? location?)
>   * file paths     (portability: File::Spec + File::Basename + ...) 
>   * file install   (portability: copy file, set file perms, etc.)
>   * query metadata (perform API calls to metadata/search server)
>   * installer lib  (all the real brains/glue for the module repo client)
>   * installer ui   (CLI and/or Readline, minimal brains, uses lib)

Too heavy weight. And honestly, I think it's backwards. Aviary should be 
a standalone "Pack" that has Parrot as its first dependency. (If you 
have Parrot installed, great, if not, it'll install it for you. It's 
pretty much what Rakudo does already.)

> * Basic Batteries modules
>   * Full versions of any modules that are limited in Core
>   * Installer add-ons:   VCS fetch/use system pkgs/full depresolve/etc.
>   * Module author tools: create/register/update/upload/etc.
>   * PIR-level tools:     disassembler/debugger/profiler/data dumper
>   * NCI tools:           parse header/manage typemap/wrap C struct/etc.
>   * Standard interfaces: TAP, DBDI, logging, ?
>   * Standard libraries:  OpenSSL, DateTime, temp dir/file, ?
> 
> * Possible Power Packs (NOTE: *EXAMPLES ONLY*, DON'T BIKESHED!)
>   * Database:     DBDs (drivers), SQL clients, per-HLL DBI variants
>   * Testing:      smoke/tinder/smolder clients, per-HLL Test::* variants
>   * Security:     SSH, GPG, libpcap, ...
>   * Unixen:       POSIX, Fcntl, Errno, ...
>   * Markup:       YAML, libxml2, Expat, DOM, SAX, ...
>   * VCS:          CVS, Subversion, git, Mercurial, ...
>   * Email:        POP, IMAP, SMTP, MIME, ...
>   * GUI:          Qt, GTK+, Wx, Tk, ...
>   * 2D Graphics:  libpng, GD, SDL, Cairo, ...
>   * 3D Graphics:  OpenGL, EGL, GLU, ...
>   * Sound:        OpenAL, Pulse Audio, JACK, ...
>   * Game Support: Require other Power Packs: Audio, 2D/3D Graphics

Details of what modules go where can come later. First step is to get 
Pack installs working. (Basic Batteries is just a small Pack.)

Aviary could include tools to make it very easy to create a Pack from a 
simple JSON file (a list of modules, and a title/description for the 
Pack). All Packs should be standalone, installing Parrot and Aviary if 
needed.

> * Misc recommendations
>   * Separate 'parrot-modules' mailing list for module creators/users.

Nothing kills a good idea faster than shoving it off on a separate 
mailing list that no one reads. If module-specific traffic ever seems to 
be overwhelming parrot-dev we can split it off. (Since we got the ticket 
traffic off parrot-dev, traffic is quite tolerable now.)

>   * Default to simple (CPAN-style) dependency resolution; upgrade to
>     full resolution and system package awareness in Basic Batteries.

Not sure what you mean. More detail?

>   * Names so far suggested for module repository network:
>     + Aviary

Love it. Very Parroty. aviary.parrot.org?

>     + CPAAN
>     + FPAN

Both would lead people to expect CPAN, which would be limiting to us, 
and frustrating to them when they find out it's not.

> Metadata Proposal
> -----------------
> 
> * Required fields:
>   * meta-spec
>     + version
>     + uri
>   * name
>   * authority
>   * version
>   * license
>     + type
>     + uri
>   * copyright_holder
>   * abstract

Good.

> * Manifest fields:
>   * files
>     + configure
>     + build
>     + test
>     + install
>       - share
>       - docs
>       - bin
>       - lib
>       - runtime

Skip the manifest, it's a pile of duplicated data that's only needed by 
the build process (by the time you start the build process, you have the 
tarball anyway).

> * Dependency fields (as { [dep_name]: [version_spec], ... }):
>   * provides
>   * conflicts
>   * requires
>     + fetch
>     + configure
>     + build
>     + test
>     + install
>     + runtime

Good.

> * Optional features fields:
>   * optional_features
>     + [feature_name]
>       - description
>       - [any/all dependency fields as needed]

Good.

> * Other optional fields:
>   * description
>   * keywords
>   * generated_by
>   * contributors
>     + authors
>     + maintainers
>     + translators
>     + testers
>     + reviewers
>   * resources
>     + source
>     + homepage
>     + bugtracker
>     + wiki
>     + repository
>       - type
>       - checkout_uri
>       - browser_uri
>       - project_uri
>     + mailinglists
>       - [list_name]
>         . address
>         . uri

Good.

> * Undecided fields:
>   * dynamic_config
>   * no_index
>   * digests
>   * signatures

We should be prepared for the format to grow and change over time, 
possibly allowing custom fields.


I don't see anything here for "standard build instructions". As in, the 
specific command-line instructions for "configure", "build", "test", and 
"install". These could allow variable substitutions from parrot_config 
(or Aviary's collected configuration information), so Rakudo's "perl 
Configure.pl" could be "@perl@ Configure.pl", while Pynie's "python 
setup.py build" could be "@python@ setup.py build", and a general 'make 
test' could be "@make@ test" (to allow for nmake, etc).

Allison