This repo contains several components:

• /metadata-api - a REST-API that serves meta data for SB's corpora, lexicons and models (mainly used by our site at spraakbanken.gu.se). For documentation see below.
• /parse/create_metashare.py - a script that creates metashare files for our Clarin repo
• /parse/gen_pids.py - a Python script that generates new PIDs (Datacite DOIs) by reading our metadata YAML-files and registering resources at Datacite. For documentation see below (after the metadata-api documentation).
• /parse/parse_yaml.py - a sript that prepares data for the REST-API
• update_metadata.sh - a shell script that runs periodically on the server and calls and starts the other components. It also handles all the git stuff as both gen_pids.py and parse_yaml.py works with our metadata repository

metadata-api

Requirements

• Python 3
• Memcached (optional)

Usage

Available API calls:

• https://ws.spraakbanken.gu.se/ws/metadata: List all resources in three dictionaries (corpora, lexicons, and models)
• https://ws.spraakbanken.gu.se/ws/metadata/corpora: List all corpora
• https://ws.spraakbanken.gu.se/ws/metadata/lexicons: List all lexicons
• https://ws.spraakbanken.gu.se/ws/metadata/models: List all models
• https://ws.spraakbanken.gu.se/ws/metadata/collections: List all collections
• https://ws.spraakbanken.gu.se/ws/metadata?resource=saldo: List one specific resource. Add long description from SVN (if available)
• https://ws.spraakbanken.gu.se/ws/metadata/list_ids: List all existing resource IDs
• https://ws.spraakbanken.gu.se/ws/metadata/check-id-availability?id=my-resource: Check if a given resource ID is free
• https://ws.spraakbanken.gu.se/ws/metadata/doc: Serve API documentation as YAML
• https://ws.spraakbanken.gu.se/ws/metadata/renew-cache: Flush cache and fill with fresh values

Installation (SBX-specific)

• Install requirements from requirements.txt, e.g. with a (virtual environment):

  python3 -m venv venv
  source venv/bin/activate
  pip install -r requirements.txt
  

• Get an initial copy of the metadata files (YAML and META-SHARE):

   git clone https://github.com/spraakbanken/metadata.git
  

• Add entry in supervisord config:

  [program:metadata]
  command=%(ENV_HOME)s/metadata-api/venv/bin/gunicorn --chdir %(ENV_HOME)s/metadata-api -b "0.0.0.0:1337" metadata_api:create_app()
  

• Set up cron job that periodically runs the update script which

  • updates the metadata files stored in git
  • runs the python script for parsing these files
  • updates the repository from GitHub and restarts the service if needed
  • creates missing META-SHARE files

  The following cron job is run on fksbwww@k2:

  # Update sb-metadata from GitHub and restart if needed
  50 * * * * cd /home/fksbwww/metadata-api && ./update_metadata.sh > /dev/null
  

Collections

A collection is a "meta" metadata entry which is used to summarize multiple resources. Collections are supplied as YAML files. The resource-IDs belonging to a collection can either be supplied as a list in the YAML (with the 'resources' key) or each resource can state which collection(s) it belongs to in its YAML (with the 'in_collections' key which holds a list of collection IDs). The size of the collection is calculated automatically. A collection may have a resource description in its YAML.

Unlisted

Resources with the attribute "unlisted": true will not be listed in the data list on the web page, but they can be accessed directly via their URL. This is used as a quick and dirty versioning system.

Successors

The successors attribute can be used for resources that have been superseded by one or more other resources (e.g. newer versions). This attribute holds a list of resource IDs.

gen_pids.py

For documentation see the code comments and the /docs directory.

Storing credentials

The Datacite login credentials are store in a .netrc file located in /home/fksbwww on the server.