Skip to main content

Source code file content

Revision: 2971

15771544 pkgrepo should have a way of removing an entire publisher (fix test)
» Project Revision History

» Checkout URL

pkg-gate / doc / catalog-v1.txt

Size: 49883 bytes, 1 line
pkg(5): image packaging system


1.  Overview

    The pkg(5) server and client catalogs currently provides a summary
    view of the packages provided by a repository: the FMRIs of each
    package, the last time the set of available packages changed, and
    the total number of packages.  The server uses this information
    for publication checks, to fulfill client requests, for search
    indexing and analysis, and to enable browser-based access to the
    repository via the BUI (Browser User Interface).  pkg(5) clients
    use this information to determine what packages are available, to
    validate user input, and to fulfill packaging operation requests.

1.1  History

    As development of the Image Packaging System has progressed, both
    the server and client have increasingly required access to more
    packaged metadata as fixes and various improvements have been
    implemented.  This has resulted in increased demand on server and
    client system resources when analyzing package metadata, and
    increased processing times as well.

    To address catalog performance issues, a client-side unified
    catalog cache was implemented, and initially contained all known
    package stems from the set of publishers defined within the image
    configuration.  The caching mechanism was then replaced, using a
    Python dict structure designed for efficient lookups of package
    information by stem or FMRI and providing an ordered list of
    versions, that was then serialized to disk.

    Recently, the caching was revised to use a custom, delta-encoded
    text format that avoided object serialization as that created an
    implicit dependency on object versions and definitions, as well as
    significant overhead in the on-disk footprint.  To improve package
    metadata performance, a new cache format was created that factored
    package manifests by the types of actions contained within, and
    then stored each type of action in a separate file for each

1.2  Challenges

    Despite past improvements, significant performance improvements
    are still needed for both the server and client when processing
    and analyzing package metadata.  The work done so far has also
    only benefited the client, leaving server performance behind.
    Specifically, the underlying catalog data, caching mecahnisms,
    and catalog retrieval operations suffer from the following

    - the catalog format used for the server and client is not
      consistent and the server uses local time instead of UTC

    - the client does not maintain a 1:1 copy of the server's catalog
      and attributes making it difficult to verify its integrity and
      complicates the logic needed to obtain updates

    - the caching mechanisms implemented are not granular enough,
      causing some operations to run slower than necessary as more
      information than is needed is loaded and processed

    - no efficient lookup mechanism exists for some of the metadata,
      causing operations such as dependency calculation to require a
      linear scan and retrieval of manifests

    - the existing caching mechanisms require clients to retrieve
      manifests for all known packages to be able to perform summary
      listings of available packages (at least 65 MiB for a new build
      of OpenSolaris) -- which is especially harmful to GUI clients
      such as packagemanager(1)

    - the existing caching mechanisms do not provide the information
      needed to determine (ahead of time) what package manifests need
      to be retrieved during packaging operations, which leaves pkg(5)
      clients unable to provide sufficient feedback to the user during
      plan creation such as number of bytes to be transferred, time
      estimates, etc.

    - the catalog operation and caching mechanisms offered by the
      depot server are not extensible, and cannot accommodate new
      metadata that may be needed to perform client operations
      without a client and server revision

    - the catalog and caching mechanisms do not account for
      future localization needs

1.3  Goals

    So then, the changes proposed within this document have the
    following goals:

    - unification of the server and client catalog format and code

    - simplification of catalog update and retrieval mechanisms

    - improved granularity and transparency in caching mechanisms
      allowing operations to only retrieve the information they need

    - reduction of resource requirements and processing time forserver
      and client

    - increase of available metadata before long-running package
      operations to enable improved progress and user feedback

    - improved extensibility of the catalog depot operation and the
      caching mechanisms used by the client

    - unification and implementation of caching mechanisms and code
      for client and server

2.  Proposed Changes

    The changes needed to accomplish the goals listed in section 1.3
    are grouped below by the type of change.  It should be noted that
    what is described in this document is dependent on an upcoming image
    and repository format versioning proposal since these changes will
    require a change to the structure of both images and repositories.

2.1 Catalog Format Changes

2.1.1  Current Catalog Format

    To better understand the proposed changes, it may first be helpful
    to understand the current catalog format and how it is composed.
    Currently, the catalog could be viewd as being composed of three

    - attrs

        The attrs file contains metadata about the catalog.  The
        server and client attrs file are text/plain, and currently
        have the following content:

         S Last-Modified: 2009-06-23T07:58:35.686485
         S prefix: CRSV
         S npkgs: 40802

        The client adds this content:
         S origin: <repository_uri>

        The Last-Modified value is an ISO-8601 date and time in server
        local time (not UTC).

    - catalog

        The server catalog file currently contains entries of this

        <type> <fmri><newline>

        Where type can be 'V' (version), 'C' (critical; not used), or
        'R' (renamed).

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

        If a destination package is not provided for 'R', then 'NULL'
        is used for the destination values.


        C pkg:/foo@0.5.11,5.11-0.111:20090507T161015Z
        V pkg:/foo@0.5.11,5.11-0.111:20090508T161015Z
        R foo 1.0:20090508T161015Z bar 1.0:20090509T161015Z
        R baz 1.0:20090508T161015Z NULL NULL

        The client catalog file contains entries of this format:

        <type> pkg <fmri_stem> <fmri_version><newline>

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

        If a destination package is not provided for 'R', then 'NULL'
        is used for the destination values.


        V pkg foo 0.5.11,5.11-0.111:20090508T161015Z

    - update log

        While not strictly a part of the catalog, the update logs serve
        as a record of changes to the catalog allowing clients to otbain
        incremental updates to a catalog instead of retrieving the
        entire catalog each time.

        It only exists on the server, and contains entries of this


        Where 'update_type' can be '+' (there were comments at one
        time referring to a '-' operation, but they were removed and
        the code for it was never implemented).

        Where 'type' can be 'V' (version), 'C' (critical; not used),
        or 'R' (renamed).

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

2.1.2  Proposed Catalog Format

    To accomplish the goals listed in section 2.1, a new catalog
    format will be adopted.  This format will be used by the client
    to store catalog data locally, regardless of the format used by
    the repository (e.g. the repository only provides older catalog
    format).  All data is assumed to be encodable using Unicode as
    the JSON format specification requires this.

    The new catalog format splits the contents of the catalog into
    multiple parts, per-locale, but treats them as a unified set.
    That is, all of the parts have a common base, but can easily be
    merged at load time if access to multiple parts is needed.

    The catalog will be composed of the following files:

    - catalog.attrs
        This file will contain a python dict structure serialized in
        JSON (JavaScript Object Notation) format.  The metadata within
        is used to describe the catalog and its contents using the
        following attributes:

            An optional dict of digest and/or cryptograhic values which
            can be used by clients to verify the integrity of the
            catalog.attrs data.  Each key should represent the name of
            the signature or digest used, and each value the signature

            The value is an ISO-8601 'basic format' date in UTC time
            indicating when the catalog was created.  This value is
            provided by the server.

            The value is an ISO-8601 'basic format' date in UTC time
            indicating when the catalog was last updated.  This value
            is provided by the server.

            An integer value indicating the total number of unique
            FMRI stems in the catalog.

            An integer value indicating the total number of unique
            FMRI versions in the catalog.

            A dict of available catalog parts.  This is to enable
            clients to quickly determine what locale-specific catalog
            data might be available to them.  Each entry contains the
            date and time a part was created and last modified.  It
            may also contain digest signature entries for the part (if
            available) so that clients can verify parts after applying
            incremental updates.

            A dict of available catalog updates.  Each entry corresponds
            to the filename of an update log named after the time the
            update occurred using an ISO-8601 'reduced accuracy basic
            format'.  Each entry also contains a last-modified date in
            ISO-8601 basic format to allow clients to determine when an
            update log was last changed without checking the repository.

            An integer value representing the version of the structure
            used within the attrs, update log, and catalog part files.


            "_SIGNATURE": {
              "sha-1": "8f5c22fd8218f7a0982d3e3fdd01e40671cb9cab"
            "created": "20050614T080000.234231Z",
            "last-modified": "20090508T161025.686485Z",
            "package-count": 40802,
            "package-version-count": 1706,
            "parts": {
              "catalog.base.C": {
                "last-modified": "20090508T161025.686485Z",
                "signature-sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45",
              "catalog.dependency.C": {
                "last-modified": "20090508T161025.686485Z",
                "signature-sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",
              "catalog.summary.C": {
                "last-modified": "20090508T161025.686485Z",
                "signature-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7",
              "catalog.summary.FR": {
                "last-modified": "20081002T080903.235212Z",
                "signature-sha-1": "d2b6cb03677c725f94c9ba551a454d56b54ea2f8",
            "updates": {
              "update.20081002T08Z.C": {
                "last-modified": "20081002T080903.235212Z",
                "signature-sha-1": "a2b6cb03277c725a94c9ba551a454d56b54ea2f8",
              "update.20090508T16Z.C": {
                "last-modified": "20090508T161025.686485Z",
                "signature-sha-1": "c2b6ca03473c725f94c8ba201a454d56b54ea2f8",
            "version": 1,

    - catalog.<part_name>.<locale_name>

        Each part of the catalog will contain a python dict structure
        serialized in JSON format.  <locale_name> is an OpenSolaris
        system locale name, and should be 'C' if the file applies to
        all locales.  Version entries for each package stem are kept
        in ascending version order to allow fast lookups by the client
        and avoid sort overhead on load.  Also note that any top-level
        entry key in the structure starting with '_' will be treated
        as metadata related to the catalog or version entry and must
        be ignored unless the client can use it.

        Finally, each catalog entry can also contain an optional set
        of digest and signature key/value pairs that can be used to
        verify the content of the related package manifest.  Clients
        must ignore any key/value pairs that are unknown to them within
        the structure.  The catalog structure can be described as

          "<optional-signature-dict": {
            "<signature-name>": "<signature-value>",
          "<publisher-prefix>": {
            "<FMRI package stem>": [
                "version": <FMRI version string>,
                "<optional-actions>": <optional-actions-data>
                "<optional-signature-name>": "<signature-value>",

    Initially, the server will offer the following catalog 'parts'.
    Each has its content based on a tradeoff between memory usage,
    load times, and bandwidth needs which depend on the client being
    used to perform packaging operations or the operation being

    - catalog.base.C
        This catalog part contains the FMRIs of the packages that the
        repository contains, and an optional digest value that can be
        used for verifying the contents of retrieved manifests.  Loading
        just this content is useful when performing basic listing opera-
        tions using the cli, or when simply checking to see if a given
        package FMRI is valid.  Note that since this information is com-
        mon to all locales, this part of the catalog is only offered for
        the 'C' locale.  An example of its contents is shown below:

          "_SIGNATURE": {
            "sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45",
                "signature-sha-1": "596f26c4fc725b486faba089071d2b3b35482114"
                "signature-sha-1": "ab6f26c4fc725b386faca089071d2b3d35482114"
                "signature-sha-1": "fe6f26c4fc124b386faca089071d2b3a35482114"
                "signature-sha-1": "696f26c4fc124b386facb089071d2b3f35482114"

    - catalog.dependency.C
        This catalog part contains the FMRIs of the packages that the
        repository contains, any 'depend' actions, and any 'set' actions
        for facets or variants.  This information is intended to be used
        during dependency calculation by install, uninstall, etc.  It is
        anticipated that package size summary information, and actions
        for set pkg.renamed and pkg.obsolete will be stored in this part
        as well when they become available.  Note that since this infor-
        mation is common to all locales, this part of the catalog is
        only offered for the 'C' locale.  An example of its contents is
        shown below:

          "_SIGNATURE": {
            "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",
                  "set value=global value=nonglobal",
                  "set name=variant.arch value=sparc value=i386",
                  "depend fmri=SUNWlibms@0.5.11-0.108 type=require",
                  "depend fmri=SUNWlibC@0.5.11-0.108 type=require",
                  "depend fmri=SUNWcsl@0.5.11-0.108 type=require"

    - catalog.summary.<locale_name>
        This catalog part contains the FMRIs of the packages that the
        repository contains and any 'set' actions (excluding those for
        facets or variants).  This information is intended to be used
        primarily by GUI clients such as packagemanager(1), or the BUI
        (Browser UI) provided by pkg.depotd(1m) for quick, efficient
        access to package metadata for listing.  An example is shown

          "_SIGNATURE": {
            "catalog-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7",
                  "set name=description value=\"DVD creation utilities\"",
                  "set name=info.classification value=org.opensolaris.category.2008:System/Media",
                  "set name=description value=\"Thunderbird localization - other 15 lang\"",
                  "set name=srcpkgs value=SUNWthunderbirdl10n-extra"

    - update.<logdate>.<locale_name>

        This file will contain a python dict structure serialized in
        JSON (JavaScript Object Notation) format.  Where <logdate> is
        a UTC date and time in  ISO-8601 'reduced accuracy basic
        format'. <locale_name> is an OpenSolaris system locale name,
        and should be 'C' if the update log applies to all locales.

        The structure of catalog update files is similar to that of
        of catalog files, with a few exceptions.  First, each version
        entry contains additional elements indicating the catalog
        operation and the time of the operation.  Second, each entry
        also contains a set of dicts keyed by catalog part name
        indicating which catalog parts the package was added to
        contents of each of these dicts is the exact contents of the
        package's catalog part entry (excluding version).

        The supported types (<op-type> in the example below) of catalog
        operations are:

        'add'       Indicates that the corresponding FMRI and metadata
                    (if present) has been added to the catalog.

        'remove'    Indicates that the corresponding FMRI has been
                    removed from the catalog.

        The structure can be described as follows:

          <optional-signature-or-signature-dict>: {
            <signature-or-signature-name>: <signature-or-signature-value>,
          <publisher-prefix>: {
            <FMRI package stem>: [
                "op-type": <type-of-operation>
                "op-time": <ISO-8601 Date and Time>
                "version": <FMRI version string>,
                "<>": <catalog-type-metadata>,

        An example update log might consist of the following:

          "_SIGNATURE": {
            "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",
                "op-type": "remove",
                "op-time": "20090218T042838Z"
                "op-type": "add",
                "op-time": "20090524T042841Z",
                "catalog.dependency.C": {
                  "actions": [
                    "depend fmri=SUNWlibms@0.5.11-0.111 type=require",
                    "depend fmri=SUNWlibC@0.5.11-0.111 type=require",
                    "depend fmri=SUNWcsl@0.5.11-0.111 type=require"
                "catalog.summary.C": {
                  "actions": [
                    "set name=description value=\"DVD creation utilities\"",
                    "set name=info.classification value=org.opensolaris.category.2008:System/Media",
                "signature-sha-1": "fe6f26c4fc124b386faca089071d2b3a35482114",

    Please note that the digest and cryptographic information is
    optional since older repositories won't have the information and
    some users of the depot software may choose to not provide it.
    For a detailed discussion on the choice of data format and a
    performance analysis, see section 3.

2.2  Server Changes

    To enable clients to retrieve the new catalog files and incremental
    updates to them, the following changes will be made:

    - The new catalog files will be stored in the <repo_dir>/catalog
      directory using the filenames described in section 2.1.2.  Any
      existing catalog files will be converted to the new format upon
      load (using writable-root if present) and the old ones removed
      (unless readonly operation is in effect).

    - Operations that modify the catalog file will be changed to write
      out all of the new catalogs only; the version 0 catalog will no
      longer be stored or used.

    - The depot server will be changed to offer an additional catalog
      operation "/catalog/1/" which will be added to the output of the
      "/versions/0/" operation as well.  It will provide a simple GET-
      based HTTP/1.1 interface for retrieving catalog and update log
      files from the server.  It will not require or use any headers
      other than those normally present within a standard HTTP/1.1
      transaction.  However, the client api will continue to provide
      the uuid, intent, and user agent headers that are provided today
      for the existing "/catalog/0/" operation.

    - The existing "/catalog/0/" operation will continue to be offered
      by the depot server for compatibility with older clients.

    - The depot server will be changed to perform a simple sanity check
      when starting to verify that the packages in the catalog are
      physically present in the repository and that the catalog attrs
      files match the catalog files.  Likewise, the update logs will
      be checked to verify that they are valid for the catalogs.  If
      any of these files are found to be not valid, a warning will be
      logged and the catalog files rewritten (using writable-root if
      applicable).  In addition, any of the corrections made will
      result in corresponding update log entries so that incremental
      updates will not be broken for existing clients.

2.3 Client Changes

    To take advantage of the new catalog format, and to improve the
    performance of clients, a number of changes will need to be made
    to the pkg.client.api and its supporting classes.  All of the
    changes proposed here should be transparent to client api

2.3.1  Image Changes

    - The image object, upon initialization, will remove the
      /var/pkg/catalog directory and its contents if possible.
      If this cannot be done (due to permissions), the client
      will continue on.  If it can be removed, a new directory
      named /var/pkg/publisher be created, and publisher objects
      will be told to store and retrieve their metadata from it.

    - Publisher objects will store their catalog data within the
      directory <meta_root>/<prefix>/catalog/.

    - Any functions contained within the image class for the
      direct storage, retrieval, updating, etc. of publisher
      metadata will be moved to the pkg.client.publisher and
      Catalog classes.

    - A new "Catalog" object reference will be added to the
      image class, which will be used to allow the api access
      to catalog metadata.  This object will allow callers to
      ask for a specific set of catalog data for an operation
      (where the allowed sets match the names of the catalogs
      described in section 2.1.2).  The data will then be
      retrieved and stored for usage by callers as needed.

    - The existing catalog caching mechanism will be removed
      completely as it has been superseded by the new catalog

    - For performance reasons, the client api will also store
      versions of each of the catalogs proposed that only
      contain entries for installed FMRIs to accelerate common
      client functions such as info, list, uninstall, etc. This
      change will also result in the obsoletion of the current
      /var/pkg/state directory and /var/pkg/pkg/<stem>/<ver>/
      installed files, which will be removed and converted
      during the image initialization process if possible.

    - All api functions will be changed to retrieve the catalog
      data they need instead of depending upon api consumers to
      do so.

2.3.2  Catalog Retrieval and Update Changes

    - If a repository only offers version 0 of the catalog format,
      then the client API will retrieve it, but transform and store
      the catalog in version 1 format using the times the server

    - If version 1 catalog data is not available, the client api will
      fallback to retrieving catalog metadata by retrieving package
      manifests (as it does today).  This will be transparent to

    - When checking for catalog updates, the api(s) will follow this
      process for version 1 catalogs when determining if a full or
      incremental update should be performed for each catalog in the

      * If the repository now offers a version 1 catalog, but did not
        do so previously, a full catalog retrieval will be performed.

      * A conditional retrieval of the catalog.attrs file will be
        performed using the last-modified date contained within it.
        If a 304 (or not modififed status) is returned, then the
        catalog will be skipped during the update process.

      * The resulting file will then be loaded and the integrity of the
        attrs file verified by omitting the '_SIGNATURE' portion of the
        data structure and using the information that was present within
        to verify the integrity of the attrs file.  If the integrity
        check fails, a transport exception will be raised.

      * If the attrs file was retrieved successfully, it will be checked
        as follows:

        - If the created date in the retrieved attrs file does not
          match the stored attrs file, a full catalog retrieval will be
          performed as the catalog has been rebuilt.  In addition, a
          warning will be provided to the client that there may be
          something wrong with the repository (packages may be missing,

        - If the created date matches, then the version in the new attrs
          file will be compared to the original, if they do not match a
          full catalog retrieval will be performed as the format of the
          catalog has changed (unless the client is unable to parse that
          format in which case an error will be raised).

        - If the version was valid, then the last modified date in the
          new catalog.attrs file will be compared to the original attrs
          file.  If the original attrs date is newer, then a full
          catalog retrieval will be performed and the user will be
          warned that there may be something wrong with the repository
          (packages may no longer be available, etc.).  If the last
          modified date in the original attrs file is the same as the
          new attrs file, then no updates are available and the catalog
          will be skipped.  If the original attrs last modified date is
          older than the new attrs last modified date, then the 'update-
          logs' property will be checked to see if there are incremental
          updates available.  

        - If the update-logs property is empty, a full catalog retrieval
          will be performed with the assumption that the repository has
          intentionally discarded all of its incremental update
          information.  If the oldest update log listed in the new attrs
          file is newer than the last modified date of the original
          attrs file, then this client has not performed an incremental
          for a period long enough that the repository no longer offers
          incremental updates for their version of the catalog, and a
          full catalog retrieval will be performed.

        - Finally, if all of the above was successful, the api will then
          start the incremental update process.

    - When attempting to determine what incremental catalog updates
      for version 1 catalogs are available, and the repository offers
      version 1 catalogs, the client api(s) will use the following

        * The modified date and time of the update log the client last
          retrieved will compared against the corresponding entry in
          catalog.attrs.  If it has not been modified, the update log
          will be skipped.  Otherwise it will be retrieved, and added
          to the incremental update queue. This is necessary since
          update logs are per-hour and a change may have occurred since
          the last time the update log was retrieved.

        * The api will then retrieve any remaining update logs listed in
          the catalog.attrs file that have a <logdate> newer than the
          last time the client's local copy of the catalog was updated.
          Each will be added to the update queue after retrieval.

        * Each update log file will then loaded and verified by omitting
          the '_SIGNATURE' portion of the structures and using the
          information that was present within it to verify the integrity
          of the update log.  If the integrity check fails, a transport
          exception will be raised.

    - When applying the queued catalog updates, the client api will
      use this process for each update log:

        * each corresponding catalog part present in the image will be
          loaded, and then any update log entries newer than the last
          modified date of the catalog (based on op-time) will be
          applied to the catalog as dicated by op-type

        * if at any point, an update log entry cannot be applied as
          directed, then a full catalog retrieval will be forced, and
          the user will be warned that something may be wrong with the
          repository (missing packages, etc.)

        * if the update log is the last in the queue for a given set of
          catalogs, then all previous ones will be removed as they are
          no longer needed

    - When attempting to verify the integrity of a full catalog part
      retrieval, the api will use this process:

      * The catalog parts will be loaded into memory and the
        '_SIGNATURE' portion of the data structure removed.

      * The api will then check the catalog.attrs file for digest
        and/or cryptographic information related to the catalog.
        If the information is present, it will then be used to
        verify the integrity of the retrieved catalog parts.

3.  Appendix

3.1  Overview

    During the development of this proposal, a number of different
    approaches to the storage and retrieval of catalog data were
    considered.  Specifically, the following formats were considered
    and/or evaluated:

    - manifest
        A pure "manifest-style" format similar to the existing package

    - JSON
        The portable JavaScript Object Notation-based format.

    Size and performance characteristics for each of these formats can
    be found in section 3.3.

3.2  Evaluations

3.2.1  manifest-style format evaluation

    Initially, the "manifest-style" format seemed promising from a
    performance and disk footprint standpoint when compared to using
    JSON.  A few variations of this format were attempted, and examples
    of this can be seen below:

    - variation 1
        pkg://,5.11-0.86:20080422T230436Z require=SUNWlang-common@0.5.11-0.86,SUNWcsl@0.5.11-0.86

    - variation 2
        depend fmri=pkg:/SUNWlang-common@0.5.11-0.86 type=require
        depend fmri=pkg:/SUNWcsl@0.5.11-0.86 type=require

    - variation 3
        After realising that variant and facet information was needed,
        and that additional attributes might need to be accounted for in
        the future, variation 3 was chosen for evaluation.

        set value=global value=nonglobal
        set name=variant.arch value=sparc value=i386
        depend fmri=SUNWcsl@0.5.11-0.106 type=require
        depend fmri=SUNWlang-common@0.5.11-0.106 type=require

3.2.2  JSON format evaluation

    When first evaluating JSON, results on x86-based systems were very
    comparable or significantly better than the manifest-based format
    from both a file size and performance perspective.  The following
    structural variations were evaluated:

    - variation 1
        Variation one attempted to combine the catalog and attrs files,
        but this approach was abandoned for simplicity and performance
        reasons in later variations.

        "attributes": {
            "id": "556599b2-aae8-4e67-94b3-c58a07dbd91b",
            "last-modified: "2009-05-08T16:10:25.686485",
            "locale": "C",
            "package-count: 40802,
            "version: 1,
        "packages": {
            "SUNWipkg": {
                "publisher": "",
            "versions": [

    - variation 2

    - variation 3
        This variation was attempted due to extreme performance issues
        that were seen on some lower-memory bandwidth SPARC systems
        when writing JSON files.  It was discovered that the simplejson
        library uses a recursive call structure for iterative encoding
        of python data structures and this does not perform well on many
        SPARC systems.
        By changing the structure to a list of lists, a decrease om
        write times of 20-30 seconds was realised.  However, this was
        less than desirable as it meant the resulting data structure
        would have to be significantly tranformed after load for use
        by the package system.


    - variation 4
        This variation was struck upon after the failure of the last
        with the attempt to have a data structure that was immediately
        useful to the packaging system after load:

      '': {
        'SUNWsongbird': [
            'depend': {
              'require': [
            'version': '0.5.11,5.11-0.99:20081002T152038Z'
        'SUNWstc': [
            'version': '0.5.11,5.11-0.106:20090131T191239Z'

    - variation 5
        The final variation is what was chosen for final evaluation for
        JSON after discussions with other team members centered around
        a key point: that the catalog is essentially an action pipeline
        for the client.  In addition, the prior variations were either
        hampered by poor serialization performance on SPARC systems or
        lacked the extensibility needed for possible future attribute
        additions to actions.
                  "set name=description value=\"DVD creation utilities\"",

3.2.3  Performance Analysis

    While a performance analysis was done for each variation during the
    evaluation process, only the results for the chosen variation are
    shown here.  Analyis was performed using a dump of the /dev repo
    for builds 118 and prior consisting of 42,565 unique FMRIs.

    Each format that was evaluated presented unique challenges.  While
    the manifest-style provided simplicity and familiarity, it became
    increasingly apparent during testing that any code that was used
    to parse and write it would have to be changed significantly each
    time changes were made to any in-memory structures that were used
    as the source.  In contrast, the JSON format made it easy to re-use
    the in-memory python structure as the same format to be written to

    The uncompressed and gzip-compressed (provided because both Apache
    and cherrypy are capable of gzip compressing requests) are shown
    below for comparison.  Of special note is the 'all' catalog shown
    below which was created to evaluate the feasbility of having a
    single catalog that provided access to all commonly needed metadata
    by combining the base, dependency, and summary catalogs proposed in
    section 2.1.2.

    Size Comparison
    Catalog     Mfst. Sz.   JSON Sz.    Mfst. CSz.  JSON CSz.
    current     2.25  MiB   -           327 KiB     -
    base        2.86  MiB   2.00  MiB   305 KiB     246 KiB
    dependency  16.44 MiB   16.45 MiB   1.4 MiB     1.4 MiB
    summary     7.58  MiB   7.36  MiB   483 KiB     475 KiB
    all         21.16 MiB   21.47 MiB   1.6 MiB     1.6 MiB

    The time needed to read and write each format is shown below for
    comparison.  Several runs for each catalog were performed to verify
    that the timings were consistent, and the load of each system was
    checked to verify that timings were not skewed.

    Base Catalog Timings
    System          Mfst. Wr.   JSON Wr.    Mfst. Rd.   JSON Rd.
    mine            0.13s       0.41s       0.19s       0.05s
    ipkg.sfbay      0.19s       0.58s       0.29s       0.08s
    kodiak.eng      0.30s       0.99s       0.37s       0.08s
    cuphead.sfbay   1.18s       3.41s       1.54s       0.33s
    jurassic.eng    1.37s       3.77s       1.31s       0.46s
    Mean            0.63s       1.83s       0.74s       0.20s

    Dependency Catalog Timings
    System          Mfst. Wr.   JSON Wr.    Mfst. Rd.   JSON Rd.
    mine            0.42s       1.06s       1.13s       0.24s
    ipkg.sfbay      0.98s       1.65s       1.70s       0.39s
    kodiak.eng      0.91s       2.61s       2.22s       0.40s
    cuphead.sfbay   6.05s       9.00s       8.57s       1.57s
    jurassic.eng    3.87s       10.46s      6.48s       2.13s
    Mean            2.45s       4.96s       4.02s       0.95s

    Summary Catalog Timings
    System          Mfst. Wr.   JSON Wr.    Mfst. Rd.   JSON Rd.
    mine            0.16s       0.78s       0.58s       0.14s
    ipkg.sfbay      0.33s       1.09s       0.86s       0.22s
    kodiak.eng      0.35s       1.90s       1.10s       0.25s
    cuphead.sfbay   2.02s       6.55s       4.41s       0.92s
    jurassic.eng    1.35s       7.24s       3.34s       1.25s
    Mean            0.84s       3.51s       2.06s       0.56s

    'all' Catalog Timings
    System          Mfst. Wr.   JSON Wr.    Mfst. Rd.   JSON Rd.
    mine            0.51s       1.22s       1.48s       0.31s
    ipkg.sfbay      1.22s       1.89s       2.30s       0.51s
    kodiak.eng      1.09s       3.05s       2.93s       0.53s
    cuphead.sfbay   7.35s       10.38s      11.15s      2.02s
    jurassic.eng    4.57s       12.20s      8.28s       2.74s
    Mean            2.95s       5.75s       5.23s       1.22s

    System Notes:
    - 'mine' is an Intel Core 2 DUO E8600 with 8GiB RAM
    - ipkg.sfbay is a dual Opteron 2218 with 16GiB RAM
    - kodiak.eng is a SPARC64-VI box with 32GiB RAM
    - cuphead.sfbay is an UltraSparc-T2 with 3GiB RAM
      (likely ldom or zone)
    - jurassic.eng is an UltraSPARC-III+ with 32GiB RAM

    From the timings seen above, it should become apparent that JSON
    serialization performance is, on average, noticeably slower when
    compared to a simple manifest-style format.  In particular, this
    is very noticeable on lower memory-bandwidth SPARC systems.
    It was discovered that the likely reason for poor serialization on
    some SPARC systems is that simplejson uses a recursive function-
    based iterative encoder that does not perform well on SPARC systems
    (due to register windows?).

    This is likely because the call stack depth for the encoder will
    match that of any python structure that it encodes.  During the
    evaluation of possible format variations, this resulted in a
    hybrid approach that combined a python dict with a simple list
    of actions with the hope that further improvements could be made
    to simplejson at some future date.  Without this approach,
    significant increases in write times were seen (20-30 seconds)
    when using a pure dict-based structure.

    Conversely though, JSON read performance, on average, is noticeably
    faster compared a manifest-style format.  In part, this is because
    more work has to be performed to transform the manifest-style format
    into an equivalent python data structure.  Notably, there is a large
    cost to sorting package versions after load (having the version data
    in ascending order is extremely useful to the client).

    Finally, a comparison of the heap size overhead (defined as the
    difference between the size of the heap before loading a catalog
    and after as measured on my x86 system) is shown for comparison

    'Heap' Overhead Comparison
    Catalog     Mfst. Sz.   JSON Sz.    Increase Sz.    Inc. %
    base        9.16  MiB   12.45 MiB   +3.29  MiB      +35.92%
    dependency  32.34 MiB   51.48 MiB   +19.14 MiB      +59.19%
    summary     16.84 MiB   27.41 MiB   +10.57 MiB      +62.76%
    all         39.87 MiB   63.88 MiB   +24.01 MiB      +60.22%

3.3  Conclusion

    When comparing the numbers alone, it seems as though the manifest-
    style format should have been chosen based solely on:

    - lower memory usage (43.6% less than JSON on average)

    - faster write times (1.71s on average compared to 4.01s on average
      for JSON)

    However, ultimately, the manifest-style format was rejected for
    reasons beyond simple numbers:

    - desire for a defined grammar and syntax

    - required maintaining custom parsing and storage code

    - not easily extensible such that if additional metadata
      was needed that a protocol or file format revision might
      be required

    - when weighing read performance vs. write performance,
      read performance was considered more important as updates
      to the catalog will happen far less freqeuntly than loads
      of package data (loads took 3.01s on average for manifest-
      style compared to 0.73s on average for JSON or about 75.75%

    Instead, the JSON format was selected for the following reasons:

    - full unicode support

    - well-defined grammar and structure

    - supported data types almost exactly mirror python's own
      native data types

    - allowed easy storage of existing action data of which
      catalogs are essentially a summarized view of

    - a python library for the parsing and writing of JSON is
      part of python 2.6+

    - JSON is easily portable to other systems and myriad
      tools are available to parse and write it

    - it is anticipated that the performance of simplejson will
      only improve over time

    As a final note, the approach of using separate catalogs for each
    set of data instead of a single, merged catalog was used to reduce
    memory usage and the amount of data that needs to be transferred
    for clients.

Please Confirm