Track provenance information for all modeling inputs

[benmwebb]

IMP currently takes as input files in a variety of formats, but doesn't care where those files originate. This becomes a problem when we come to publish a modeling study and deposit the files (e.g. at PDB-dev). It's a lot of work to backtrack and try to figure out where such files came from. It would be much simpler if IMP tracked this information from day one, reading it in some standardized way from the files themselves (or the Python script), storing it in the Model, and also storing it in RMF files.

Since this is prerequisite information for outputting mmCIF files, solving this issue would be a step towards addressing #968. Much of this information is currently stored outside of the Model, mostly in PMI 1 data structures, and so currently outputting mmCIF requires PMI 1.

Only input atomic models are explicitly considered here but similar considerations should apply to restraints (e.g. where an EM map comes from), sequences (e.g. uniprot identifier), etc. (More generally, any transformation of the model, such as sampling, filtering or clustering, should also be recorded.)

Input files

• All input PDB files should contain suitable headers identifying their source.
  • "Official PDB structures": we can simply keep the existing PDB headers.
  • Modeller models: need to add a header pointing to its ModelArchive or PMP ID, or add headers to provide similar metadata (e.g. a path to the alignment file used)
  • Other comparative models (e.g. Phyre2) need similar header information.
  • Derived models (e.g. an experimental model that has been rotated and translated) need a header to point to the original model with an explanation of what was done; X-ray example; comparative modeling example
• Headers should be parsed by atom::read_pdb
• Files without headers should result in a warning; in future this can be upgraded to a hard error.
• Files that come from multiple sources (e.g. two crystal structures and a comparative model docked together) should not be allowed. Such files should be split into their constituents and suitable headers added to each.
• Other input files should contain similar custom headers as appropriate. Where the file format doesn't permit this (e.g. .mrc or .pgm files) the metadata will need to be stored somewhere else - one solution would be an accompanying JSON file (e.g. foo.mrc is described by foo.mrc.json) with domain-specific metadata.

Storage in Model

• atom::read_pdb should add suitable decorators to the created Hierarchy particles to identify their source. atom::StructureSource (see Add a Source decorator #894) is one example, although more data should be stored here (e.g. path to the file, PDB ID, version, descriptive text).
• Functions like atom::create_simplified_along_backbone and PMI's generation of initial models should copy or otherwise preserve this information in the newly-generated hierarchies, while adding additional information about the simplification applied (e.g. the resolution).
• Add a core::ProvenanceHierarchy decorator to track a separate hierarchy from the atom::Hierarchy. The root of this tree is the current state of the object, while children are inputs or previous states (and so will be both core::ProvenanceHierarchy and some other decorator such as StructureSource). A particle can be decorated as both core::ProvenanceHierarchy and atom::Hierarchy. Example hierarchies include:
  • System (atom::Hierarchy), also core::ProvenanceHierarchy root
    • Citation foo et. al
    • Clustered with k-means from
      • Ensemble 1 stored in output.1/rmfs/0.rmf3
        • Generated by MD/MC with
          • EM2D restraint using image foo.pgm
          • Crosslinks read from foo.csv
      • Ensemble 2 stored in output.2/rmfs/0.rmf3
        • ...
  • Chain (atom::Hierarchy), also core::ProvenanceHierarchy root
    • Read from PDB file foo.pdb, chain A, heavy atoms only
      • Built from template 1xyzA with alignment foo.ali
• core::ProvenanceHierarchy should be static (i.e. the same for all frames in a trajectory) so that it doesn't need to be updated during a simulation, and can be stored efficiently in an RMF file. Care also needs to be taken to avoid unnecessary duplication (e.g. each ensemble contributing to a cluster likely has the exact same set of inputs).
• Functions that change the state of the model (such as samplers, filters, or clustering algorithms) should add to the provenance hierarchy appropriately.

Storage in RMF

• Support for any new decorators should be added to RMF, so that generated output files incorporate this information.
• Code that reads and writes RMF files (e.g. clustering in PMI) should preserve this information.

[benmwebb]

For tracking provenance of most experimental information, some additional information needs to be stored in the RMF file, namely the set of restraints, which particles they act on, and which restraints were used in each sampling step.

Proposal: RMF already stores basic information about decomposed restraints. Make each set of decomposed restraints children of the 'real' restraint, which holds serialized information on the restraint itself (e.g. filename where the EM map was read from, cross correlation information, total score). The SampleProvenance decorator then contains an RMF Alias node child for each restraint used in that sampling. This information is already stored in IMP (partly in the Model, and partly in the ScoringFunction.)