Archive versions

The structure of QIIME 2 archives has evolved as QIIME 2 has been developed. This section describes each historical version of the QIIME 2 Archive format, and may be useful to interface developers whose code depends on guarantees made by that format (source code).

Version-agnostic format guarantees

Though there is significant variability in the format of QIIME 2 Archives across archive versions, all archive versions share some common traits.

These shared characteristics, defined in the _Archive class in qiime2/core/archive/archiver.py, must be consistent across all formats over time as they allow archive versions to be checked, and archives with different formats to be dispatched to the appropriate version-specific parsers.

All QIIME 2 Archives have:

• a directory named with the Archive UUID, directly under the archive root at /<UUID>/, and

• a file /<UUID>/VERSION within that directory, formatted as shown below.

Fig. 7 illustrates the Archive file system.

Fig. 7 Box and arrow diagram of the guaranteed components of an archive.

The VERSION file format is described above.

Archive Version 0

Version 0 was the original QIIME 2 Archive format, and there aren’t many V0 Archives “in the wild”. V0 Archives were produced by alpha versions of the QIIME 2 framework, and were superseded in framework version 2.0.6 on 24 October 2016.

• Result data files are written in the directory /<UUID>/data

• Result UUID, semantic type, and format information are saved in /<UUID>/metadata.yaml.

• The ArchiveFormat class in v0.py offers convenience methods for loading and parsing metadata.yaml files.

Fig. 8 illustrates the format of a Version 0 Archive.

Fig. 8 Box and arrow diagram of a v0 archive.

Archive Version 1

Released in QIIME 2 version 2.0.6, commit bdc8aed, Version 1 Archives introduce decentralized provenance tracking to QIIME 2. ArchiveFormat V1 inherits all traits of v0, modifying its __init__() and write() methods only to add provenance capture.

Note

All ArchiveFormat versions subclass their predecessor. For example, the ArchiveFormat in v1.py inherits from the ArchiveFormat in v0.py, etc. This makes it easier for humans to interpret the version history.

Provenance data is stored in the directory /<UUID>/provenance/. Specifically, metadata.yaml, action.yaml and VERSION files are captured for the current Result and each of its ancestors. Each Result’s action.yaml file and associated data artifacts (e.g. sample metadata) are stored in an action directory alongside that Result’s VERSION and metadata.yaml. Considered together, we can describe these as “provenance files”. This structure is illustrated in Fig. 9.

Fig. 9 Box and arrow diagram of the provenance files in a v1 archive. The blue “multiple files” icon represents all provenance files associated with a single action (e.g. the current action, or one of its ancestors).

The structure of V1 Archives as a whole is illustrated in Fig. 10.

Fig. 10 Box and arrow diagram of a v1 archive.

Provenance files for the current Result are stored in /<UUID>/provenance/. Provenance files for each ancestor Result are stored in directory at /<root_UUID>/provenance/artifacts/<ancestor_UUID>/.

Note

V0 Archives do not capture provenance data. As a result, if a V0 artifact is an ancestor to a V1 (or greater) artifact, it is possible for the action.yaml to list Artifact UUIDs which are not present in the artifacts directory.

Archive Version 2

Released across QIIME 2 versions 2017.9 (changelog) and 2017.10 (changelog), the directory structure of this format is identical to v1, but the action.yaml file has changed.

In commit 4389a0b the Version 2 ArchiveFormat adds an output-name key to the action section of action.yaml (unless the action type is import), assigning it the output name registered to the relevant action. Prior to this change, if one action returned multiple artifacts of the same artifact class, it was not possible to differentiate between them using provenance alone.

In commit e072706, it adds provenance support for Pipelines, adding the alias-of key to the action section of action.yaml.

Archive Version 3

Released in QIIME 2 version 2017.12 (changelog), commit 684b8b7, the directory structure of this format is identical to v1 and v2.

With this release, QIIME 2 Actions are able to take variadic arguments, allowing users to pass collections of Artifacts (Lists and Sets). A YAML representer has been added so that action.yaml can represent Sets of Artifact inputs. These will show up in action.yaml as custom !set tags.

Archive Version 4

Released in QIIME 2 version 2018.4 (changelog), commit 00a294c, this format adds citations to the directory format, adds a transformers section to action.yaml, and aligns the structure of environment:framework (also in action.yaml) to match the structure of environment::plugins::<some_plugin>.

Whenever an Action is run, its registered citations are captured. When saved, they are written to a citations.bib file inside the Archive’s provenance directory. Citations for all of the current Result’s ancestors are stored in their respective directories (e.g. /<root_UUID>/provenance/artifacts/<ancestor_UUID>/citations.bib).

Fig. 11 Box and arrow diagram of the provenance files in a v4 archive.

The overall directory structure remains identical to a v1 archive (Fig. 9).

Result-specific citation tags are also written to the transformers and environment sections of the action.yaml files, for the current Result and for all ancestors with registered citations. A new custom !cite '<citation key>' tag is use to support this in YAML.

A transformers section is added between the action and environment sections of action.yaml. Because Pipelines do not use transformers, transformers will be recorded only for Methods, Visualizers, and when importing data (such as with qiime tools import). It looks like the following:

transformers:
inputs:
    demultiplexed_seqs:
    -   from: SingleLanePerSamplePairedEndFastqDirFmt
        to: SingleLanePerSamplePairedEndFastqDirFmt
output:
-   from: q2_types.feature_data._transformer:DNAIterator
    to: DNASequencesDirectoryFormat
    plugin: !ref 'environment:plugins:types'

environment::framework was previously only a version string, and is now structured identically to each plugin action’s software_entry, with version, website, and citation sections:

framework:
    version: 2019.10.0
    website: https://qiime2.org
    citations:
    - !cite 'framework|qiime2:2019.10.0|0'
plugins:
    fragment-insertion:
        version: 2019.10.0
        website: https://github.com/qiime2/q2-fragment-insertion
        citations:
        - !cite 'plugin|fragment-insertion:2019.10.0|0'
        ...

Archive Version 5

Released in QIIME 2 version 2018.11 (changelog) commit f95f324, this format version adds archive checksums to the directory structure.

A new, md5sum-formatted checksum file has been added at /<root_UUID>/checksums.md5, with one md5sum and one filename on each line. For a more detailed specification, see the QIIME 2 Pull Request #414.

checksums.md looks like the following:

5a7118c14fd1bacc957ddf01e61491b7  VERSION
333fd63a2b4a102e58e364f37cd98b74  metadata.yaml
4373b96f26689f78889caeb1fbb94090  data/faith_pd-cat1.jsonp

...

7a40cff7855daffa28d4082194bdf60e  provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/metadata.yaml
ae0d0e26da5b84a6c0722148789c51e0  provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/action/action.yaml

Fig. 12 illustrates the V5 Archive.

Fig. 12 Box and arrow diagram of a v5 archive.

Archive Version 6

Full documentation for Archive Version 6 is pending. The source code can be found here.

Archive Version 7

Archive Version 7 development is currently being planned.