DataHQ Spec Store

An API server for managing a Source Spec Registry

Quick start

Clone the repo and install

make install

Run tests

make test

Run server

python server.py

Env Vars

• DATABASE_URL: A SQLAlchemy compatible database connection string (where registry is stored)
• AUTH_SERVER: The domain name for the authentication server
• DPP_URL: URL for the datapackage pipelines service (e.g. http://host:post/)

API

Status

/source/{owner}/{dataset-id}/{revision-number}

Note: Also, you can get info about latest and latest successful revisions by hitting following endpoints

• latest - /source/{owner}/{dataset-id}/latest
• successful - /source/{owner}/{dataset-id}/successful

Method

GET

Response

{
  "id": "<revision-id>",
  "spec_contents": <source-specifications>,
  "modified": <last-modified>,
  "state": <QUEUED|INPROGRESS|SUCCEEDED|FAILED>,
  "logs": <full-logs>,
  "error_log": [ <error-log-lines> ],
  "stats": {
    "bytes": <number>,
    "count_of_rows": <number>,
    "dataset_name": <string>,
    "hash": <datapackage-hash>
  },
  "pipelines": {
    "<pipeline-id-1>": {
      "title": "Creating CSV",
      "status": "SUCCEEDED",
      "stats": null,
      "error_log": []
    },
    "<pipeline-id-2>": {
      "title": "Creating JSON",
      "status": "INPROGRESS",
      "stats": {},
      "error_log": []
    },
    "<pipeline-id-3>": {
      "title": "Creating ZIP",
      "status": "FAILED",
      "stats": {},
      "error_log": [
        'error',
        'logs'
      ]
    }
  }
}

state definition:

• QUEUED: Flow created but not running
• INPROGRESS: Flow is running
• SUCCEEDED: Finished successfully
• FAILED: Failed to run

Upload

/source/upload

Method

POST

Headers

• Auth-Token - permission token (received from conductor)
• Content-type - application/json

Body

A valid spec in JSON form. You can find example Flow-Spec in README of planer API

Response

{
  "success": true,
  "dataset_id": "<dataset-identifier>",
  "flow_id": "<dataset-identifier-with-revision-number>",
  "errors": [
      "<error-message>"
  ]
}

Update

/source/update

Method

POST

Body

Payload in JSON form.

{
  "pipeline": "<pipeline-id>",
  "event": "queue/start/progress/finish",
  "success": true/false (when applicable),
  "errors": [list-of-errors, when applicable]
}

Response

{
  "success": success/pending/fail,
  "id": "<identifier>"
  "errors": [
      "<error-message>"
  ]
}