Skip to content

Latest commit

 

History

History
48 lines (38 loc) · 3.43 KB

ARTIFACT.md

File metadata and controls

48 lines (38 loc) · 3.43 KB
Raw

OMMX Artifact Design

OMMX Artifact is a specification for packaging and distributing OMMX Messages and associated metadata over the OCI Artifact defined in OCI Image specification 1.1.0 and OCI Distribution specification 1.1.0.

Metadata and Reference

The problem that metadata should solve is the reference between data.

  • Even if the schema of the data is defined, the moment it is saved to a file or DB and separated from the application, the relationship between the data is lost.

    • For example, within the solver process, it is clear which Instance the Solution is for, but after saving to a file or DB, it is no longer apparent.

  • Wouldn't it be better to always save the Solution and Instance together in a file?

    • This simply increases the amount of data to be saved. Considering cases where many Solutions are generated by changing parameters for a single Instance, saving all Instances is inefficient. Moreover, as the amount of data increases, the overhead of exchanging that data also becomes significant.

  • Therefore, a reference mechanism is introduced.

    • Reference refers to sufficient information to obtain the data even without the actual data.
    • If the data is not shared between machines, the path in the file system can serve as a reference, but if it is shared over the Internet, a URI is needed. Introducing a reference function is closely tied to how to retain and obtain the data.

Specification

OMMX Artifact is an OCI Artifact of media type application/org.ommx.v1.artifact. OCI Artifact is represented as an OCI Image manifest. OMMX Artifact is a collection of config, layers, and annotations.

  • config is a JSON blob with the following media types:
    • application/org.ommx.v1.config+json
  • layers consists of the following blobs:
    • application/org.ommx.v1.solution blob with the following annotations:
      • org.ommx.v1.solution.instance: (digest) The corresponding instance of the solution
      • org.ommx.v1.solution.solver: (JSON) The solver information which generated the solution as a JSON
      • org.ommx.v1.solution.parameters: (JSON) Solver parameters used to generate the solution as a JSON
      • org.ommx.v1.solution.start: (RFC3339) The start time of the solution as a RFC3339 string
      • org.ommx.v1.solution.end: (RFC3339) The end time of the solution as a RFC3339 string
    • application/org.ommx.v1.instance blob with the following annotations:
      • org.ommx.v1.instance.title: (Free string) The title of this instance
      • org.ommx.v1.instance.created: (RFC3339) When this instance was created
    • application/vnd.numpy: NumPy's ndarray with NPY format
    • application/vnd.apache.parquet: DataFrame with Parquet format
    • And other blobs with appropriate media types. The media type SHOULD be registered in the IANA media type registry.
      • Users can add arbitrary annotation to arbitrary layer. org.ommx.user. prefix is reserved for user-defined annotations.
  • Annotations in manifest:
    • TBA

Note that other annotations listed above are also allowed. The key may not start with org.ommx.v1., but must be a valid reverse domain name as specified by OCI specification.