[DRAFT] Parrot module ecosystem

Geoffrey Broadwell geoff at broadwell.org
Sun Aug 9 20:58:59 UTC 2009

To any recipients not on parrot-dev or #perl6:  The following message
gives some necessary background:


Later, we had this discussion on #parrot:


As for this message:

This is a VERY ROUGH OUTLINE of my proposal for the design of the Parrot
module ecosystem.  I'm looking for any and all comments, and I expect
many changes before we reach even rough consensus.  Please feel free to
circulate this to people who understand the problem space better than
I do!  I can use all the input I can get.

Note that there are some items explicitly marked 'undecided'.  I thought
about these enough to realize I just don't know enough yet to make a
decent recommendation, so input on these is especially appreciated.

Thanks, everyone!




* General
  * Distributions should be dead easy for module authors to create,
    and for users to install.
  * We can create a centralized metadata store, but do not want to
    build and manage a module distribution network ...
  * However it should be possible for another group to do so.

* Toolchain
  * Basic tools can assume Parrot and the core modules are working,
    but require no other dependencies internally.
  * All external tools needed to download/build/install modules will
    be specified in the module metadata.
  * Tools should be easy to configure.
  * Tools should attempt to auto-configure as much as possible.
  * Tools must properly handle the difference between user-local,
    site-local, and vendor-installed modules.

* Metadata
  * Simple, extensible format.
  * Unicode and case-retaining.
  * Must include its own spec version.
  * Sufficient for automated programs to create system packages
    (DEB, RPM, etc.).
  * Separate static v. configure-discovered v. hand-edited metadata.
    Separate files?
  * Includes fetch, configure, build, test, install, and runtime
  * Should be able to track author, mailing list, bug email/bug URI,
    wiki, homepage, source repository, etc.
  * Allows disambiguation as per Perl 6 module spec (authorities,
    versions, authors, etc.).
  * Specifies rules for dependency string parsing/interpretation.


* Overview
  * Parrot community builds a module metadata search system.
  * This search system gathers metadata from various sources, and
    allows users to query it via web browser or API, but does not
    itself store the actual modules.
  * Once found, modules can be fetched from many possible sources,
    including VCS repositories, FTP mirrors, etc.
  * Parrot team will need to standardize module metadata, provide
    the libraries and tools necessary to use the search system,
    provide guidelines for extending the toolchain, and mentor the
    growth of the ecosystem until it stands on its own.

* Metadata format
  * Served metadata container is gzip'ed tarball (.tar.gz? .tgz?).
  * Core metadata is in META.json at top level of container.
  * Container includes copies of special files (e.g. README).
  * Format for specifying non-metadata-only build scripts undecided.
  * Integrity check / authentication methods undecided.
    * Probably at least md5sum and sha1sum for source tarballs,
      but what about when pulling from raw VCS repo?

* Core modules
  * parrot config  (already exists -- config.pir)
  * HTTP client    (at least GET, with redirect and proxy support)
  * zlib           (at least decompress)
  * tar            (at least extract)
  * JSON           (at least parse)
  * version spec   (at least parse and compare)
  * library probe  (shared library info: present? version? location?)
  * file paths     (portability: File::Spec + File::Basename + ...) 
  * file install   (portability: copy file, set file perms, etc.)
  * query metadata (perform API calls to metadata/search server)
  * installer lib  (all the real brains/glue for the module repo client)
  * installer ui   (CLI and/or Readline, minimal brains, uses lib)

* Basic Batteries modules
  * Full versions of any modules that are limited in Core
  * Installer add-ons:   VCS fetch/use system pkgs/full depresolve/etc.
  * Module author tools: create/register/update/upload/etc.
  * PIR-level tools:     disassembler/debugger/profiler/data dumper
  * NCI tools:           parse header/manage typemap/wrap C struct/etc.
  * Standard interfaces: TAP, DBDI, logging, ?
  * Standard libraries:  OpenSSL, DateTime, temp dir/file, ?

* Possible Power Packs (NOTE: *EXAMPLES ONLY*, DON'T BIKESHED!)
  * Database:     DBDs (drivers), SQL clients, per-HLL DBI variants
  * Testing:      smoke/tinder/smolder clients, per-HLL Test::* variants
  * Security:     SSH, GPG, libpcap, ...
  * Unixen:       POSIX, Fcntl, Errno, ...
  * Markup:       YAML, libxml2, Expat, DOM, SAX, ...
  * VCS:          CVS, Subversion, git, Mercurial, ...
  * Email:        POP, IMAP, SMTP, MIME, ...
  * GUI:          Qt, GTK+, Wx, Tk, ...
  * 2D Graphics:  libpng, GD, SDL, Cairo, ...
  * 3D Graphics:  OpenGL, EGL, GLU, ...
  * Sound:        OpenAL, Pulse Audio, JACK, ...
  * Game Support: Require other Power Packs: Audio, 2D/3D Graphics

* Misc recommendations
  * Separate 'parrot-modules' mailing list for module creators/users.
  * Default to simple (CPAN-style) dependency resolution; upgrade to
    full resolution and system package awareness in Basic Batteries.
  * Names so far suggested for module repository network:
    + Aviary
    + CPAAN
    + FPAN

Metadata Proposal

* Required fields:
  * meta-spec
    + version
    + uri
  * name
  * authority
  * version
  * license
    + type
    + uri
  * copyright_holder
  * abstract

* Manifest fields:
  * files
    + configure
    + build
    + test
    + install
      - share
      - docs
      - bin
      - lib
      - runtime

* Dependency fields (as { [dep_name]: [version_spec], ... }):
  * provides
  * conflicts
  * requires
    + fetch
    + configure
    + build
    + test
    + install
    + runtime

* Optional features fields:
  * optional_features
    + [feature_name]
      - description
      - [any/all dependency fields as needed]

* Other optional fields:
  * description
  * keywords
  * generated_by
  * contributors
    + authors
    + maintainers
    + translators
    + testers
    + reviewers
  * resources
    + source
    + homepage
    + bugtracker
    + wiki
    + repository
      - type
      - checkout_uri
      - browser_uri
      - project_uri
    + mailinglists
      - [list_name]
        . address
        . uri

* Undecided fields:
  * dynamic_config
  * no_index
  * digests
  * signatures

More information about the parrot-dev mailing list