Skip to content

Latest commit

 

History

History
394 lines (291 loc) · 20 KB

README.md

File metadata and controls

394 lines (291 loc) · 20 KB
Raw

PyPI version DOI Build Status

grlc, the git repository linked data API constructor, automatically builds Web APIs using shared SPARQL queries. http://grlc.io/

If you use grlc in your work, please cite it as:

@InProceedings{merono2016grlc,
 author = {Mero{\~{n}}o-Pe{\~{n}}uela, Albert and Hoekstra, Rinke},
 title = {{grlc Makes GitHub Taste Like Linked Data APIs}},
 booktitle = {The Semantic Web: ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 -- June 2,  2016},
 year = {2016},
 publisher = {Springer},
 pages = {342--353},
 isbn = {978-3-319-47602-5},
 doi = {10.1007/978-3-319-47602-5_48}
}

What is grlc?

grlc is a lightweight server that takes SPARQL queries (stored in a GitHub repository, in your local filesystem, or listed in a URL), and translates them to Linked Data Web APIs. This enables universal access to Linked Data. Users are not required to know SPARQL to query their data, but instead can access a web API.

Quick tutorial

For a quick usage tutorial check out our wiki walkthrough and list of features.

Usage

grlc assumes that you have a collection of SPARQL queries as .rq files (like this). grlc will create one API operation for each SPARQL query/.rq file in the collection.

Your queries can add API parameters to each operation by using the parameter mapping syntax. This allows your query to define query variables which will be mapped to API parameters for your API operation (see here for an example).

Your queries can include special decorators to add extra functionality to your API.

Query location

grlc can load your query collection from different locations: from a GitHub repository (api-git), from local storage (api-local), and from a specification file (api-url). Each type of location has specific features and is accessible via different paths. However all location types produce the same beautiful APIs.

From a GitHub repository

API path: http://grlc-server/api-git/<user>/<repo>

grlc can build an API from any Github repository, specified by the GitHub user name of the owner (<user>) and repository name (<repo>).

For example, assuming your queries are stored on a Github repo: https://github.com/CLARIAH/grlc-queries/, point your browser to the following location http://grlc.io/api-git/CLARIAH/grlc-queries/

grlc can make use of git's version control mechanism to generate an API based on a specific version of queries in the repository. This can be done by including the commit sha in the URL path (http://grlc-server/api-git/<user>/<repo>/commit/<sha>), for example: http://grlc.io/api-git/CLARIAH/grlc-queries/commit/79ceef2ee814a12e2ec572ffaa2f8212a22bae23

grlc can also use a subdirectory inside your Github repo. This can be done by including a subdirectory in the URL path (http://grlc-server/api-git/<user>/<repo>/subdir/<subdir>).

From local storage

API path: http://grlc-server/api-local/

grlc can generate an API from a local directory in the computer where your grlc server runs. You can configure the location of this directory in your grlc server configuration file. See also how to install and run your own grlc instance.

When the API is generated from a local directory, API information can be loaded from a configuration file in that folder. This file must be called local-api-config.ini and it has the following format:

[repo_info]
repo_title = Some title
api_description = Description of my API
contact_name = My name
contact_url = https://mypage/
licence_url = https://mylicence/

From a specification file

API path: http://grlc-server/api-url/?specUrl=<specUrl>

grlc can generate an API from a yaml specification file accessible on the web.

For example, assuming your queries are listed on spec file: https://raw.githubusercontent.com/CLARIAH/grlc-queries/master/urls.yml, point your browser to the following location http://grlc.io/api-url?specUrl=https://raw.githubusercontent.com/CLARIAH/grlc-queries/master/urls.yml

Specification file syntax

A grlc API specification file is a YAML file which includes the necessary information to create a grlc API, most importantly a list of URLs to decorated and HTTP-dereferenceable SPARQL queries. This file should contain the following fields

  • title: Title of my API
  • description: API description
  • contact: Contact details of the API owner. This should include the name and url properties.
  • licence: A URL pointing to the licence file for the API.
  • queries: A list of URLs of SPARQL queries (with header decorators).

For example:

title: Title of my API
description: Description of my API
contact:
  name: Contact Name
  url: https://www.mywebsite.org
licence: http://example.org/licence.html
queries:
  - https://www.mywebsite.org/query1.rq
  - https://www.mywebsite.org/query2.rq
  - https://www.otherwebsite.org/query3.rq

grlc generated API

The API paths of all location types point to the generated swagger-ui style API documentation. On the API documentation page, you can explore available API calls and execute individual API calls.

You can also view the swagger spec of your API, by visiting <API-path>/swagger, for example: http://grlc.io/api-git/CLARIAH/grlc-queries/swagger

grlc query execution

When you call an API endpoint, grlc executes the SPARQL query for that endpoint by combining supplied parameters and decorators.

There are 4 options to specify your own endpoint:

  • Add a sparql_endpoint on your config.ini
  • Add a endpoint parameter to your request: 'http://grlc.io/user/repo/query?endpoint=http://sparql-endpoint/'. You can add a #+ endpoint_in_url: False decorator if you DO NOT want to see the endpoint parameter in the swagger-ui of your API.
  • Add the #+ endpoint: decorator.
  • Add the URL of the endpoint on a single line in an endpoint.txt file within the GitHub repository that contains the queries.

The endpoint call will return the result of executing the query as a json representation of rdflib.query.QueryResult (for other result formats, you can use content negotiation via HTTP Accept headers). For json responses, the schema of the response can be modified by using the #+ transform: decorator.

Decorator syntax

Special decorators are available to make your swagger-ui look nicer and to increase functionality. These are provided as comments at the start of your query file, making it still syntactically valid SPARQL. All decorators start with #+ , for example:

#+ decorator_1: decorator value
#+ decorator_1: decorator value

SELECT * WHERE {
  ?s ?p ?o .
}

The following is a list of available decorators and their function:

summary

Creates a summary of your query/operation. This is shown next to your operation name in the swagger-ui.

Syntax:

#+ summary: This is the summary of my query/operation

Example query and the equivalent API operation.

description

Creates a description of your query/operation. This is shown as the description of your operation in the swagger-ui.

Syntax:

#+ description: Extended description of my query/operation.

Example query and the equivalent API operation.

endpoint

Specifies a query-specific endpoint.

Syntax:

#+ endpoint: http://example.com/sparql

Example query and the equivalent API operation.

pagination

Paginates the results in groups of (for example) 100. Links to previous, next, first, and last result pages are provided as HTTP response headers to avoid polluting the payload (see details here)

Syntax:

#+ pagination: 100

Example query and the equivalent API operation.

method

Indicates the HTTP request method (GET and POST are supported).

Syntax:

#+ method: GET

Example query and the equivalent API operation.

tags

Assign tags to your query/operation. Query/operations with the same tag are grouped together in the swagger-ui.

Syntax:

#+ tags:
#+   - firstTag
#+   - secondTag

Example query and the equivalent API operation.

defaults

Set the default value in the swagger-ui for a specific parameter in the query.

Syntax:

#+ defaults:
#+   - param_name: default_value

Example query and the equivalent API operation.

enumerate

Indicates which parameters of your query/operation should get enumerations (and get dropdown menus in the swagger-ui) using the given values from the SPARQL endpoint. The values for each enumeration variable can also be specified into the query decorators to save endpoint requests and speed up the API generation.

Syntax:

#+ enumerate:
#+   - var1:
#+     - value1
#+     - value2

Example query and the equivalent API operation.

Notice that these should be plain variable names without SPARQL/BASIL conventions (so var1 instead of ?_var1_iri)

endpoint_in_url

Allows/disallows the endpoint parameter from being provided as a URL parameter (allowed by default).

Syntax:

#+ endpoint_in_url: False

Example query and the equivalent API operation.

transform

Allows query results to be converted to the specified JSON structure, by using SPARQLTransformer syntax. Notice that the response content type must be set to application/json for the transformation to take effect.

Syntax:

#+ transform: {
#+     "key": "?p",
#+     "value": "?o",
#+     "$anchor": "key"
#+   }

Example query and the equivalent API operation.

Example APIs

Check these out:

You'll find the sources of these and many more in GitHub

Use this GitHub search to see examples from other grlc users.

Install and run

You can use grlc in different ways:

More details for each of these options are given below.

grlc.io

The easiest way to use grlc is by visiting grlc.io and using this service to convert SPARQL queries into a RESTful API. Your queries can be stored on a github repo or can be listed on a specification file.

Docker

To run grlc via docker, you'll need a working installation of docker. To deploy grlc, just pull the latest image from Docker hub. :

docker run -it --rm -p 8088:80 clariah/grlc

The docker image allows you to setup several environment variable such as GRLC_SERVER_NAME GRLC_GITHUB_ACCESS_TOKEN and GRLC_SPARQL_ENDPOINT:

docker run -it --rm -p 8088:80 -e GRLC_SERVER_NAME=grlc.io -e GRLC_GITHUB_ACCESS_TOKEN=xxx -e GRLC_SPARQL_ENDPOINT=http://dbpedia.org/sparql -e DEBUG=true clariah/grlc

Pip

If you want to run grlc locally or use it as a library, you can install grlc on your machine. Grlc is registered in PyPi so you can install it using pip.

Prerequisites

grlc has the following requirements:

  • Python3
  • development files (depending on your OS):
sudo apt-get install libevent-dev python-all-dev

pip install

Once the base requirements are satisfied, you can install grlc like this:

pip install grlc

Once grlc is installed, you have several options:

Standalone server

grlc includes a command line tool which you can use to start your own grlc server:

grlc-server

Using a WSGI server

You can run grlc using a WSGI server such as gunicorn as follows:

gunicorn grlc.server:app

If you want to use your own gunicorn configuration, for example gunicorn_config.py:

workers = 5
worker_class = 'gevent'
bind = '0.0.0.0:8088'

Then you can run it as:

gunicorn -c gunicorn_config.py grlc.server:app

Note: Since gunicorn does not work under Windows, you can use waitress instead:

waitress-serve --port=8088 grlc.server:app

If you want to run grlc at system boot as a service, you can find example upstart scripts at upstart/

grlc library

You can use grlc as a library directly from your own python script. See the usage example to find out more.

grlc server configuration

Regardless of how you are running your grlc server, you will need to configure it using the config.ini file. Have a look at the example config file to see how it this file is structured.

The configuration file contains the following variables:

  • github_access_token access token to communicate with Github API.
  • local_sparql_dir local storage directory where local queries are located.
  • server_name name of the server (e.g. grlc.io)
  • sparql_endpoint default SPARQL endpoint
  • user and password SPARQL endpoint default authentication (if required, specify 'none' if not required)
  • debug enable debug level logging.
GitHub access token

In order for grlc to communicate with GitHub, you'll need to tell grlc what your access token is:

  1. Get a GitHub personal access token. In your GitHub's profile page, go to Settings, then Developer settings, Personal access tokens, and Generate new token
  2. You'll get an access token string, copy it and save it somewhere safe (GitHub won't let you see it again!)
  3. Edit your config.ini or docker-compose.yml as value of the environment variable GRLC_GITHUB_ACCESS_TOKEN.

Contribute!

grlc needs you to continue bringing Semantic Web content to developers, applications and users. No matter if you are just a curious user, a developer, or a researcher; there are many ways in which you can contribute:

  • File in bug reports
  • Request new features
  • Set up your own environment and start hacking

Check our contributing guidelines for these and more, and join us today!

If you cannot code, that's no problem! There's still plenty you can contribute:

  • Share your experience at using grlc in Twitter (mention the handle @grlcldapi)
  • If you are good with HTML/CSS, let us know

Related tools

  • SPARQL2Git is a Web interface for editing SPARQL queries and saving them in GitHub as grlc APIs.
  • grlcR is a package for R that brings Linked Data into your R environment easily through grlc.
  • Hay's tools lists grlc as a Wikimedia-related tool :-)

This is what grlc users are saying

Quotes from grlc users:

A cool project that can convert a random SPARQL endpoint into an OpenAPI endpoint

It enables us to quickly integrate any new API requirements in a matter of seconds, without having to worry about configuration or deployment of the system

You can store your SPARQL queries on GitHub and then you can run your queries on your favourite programming language (Python, Javascript, etc.) using a Web API (including swagger documentation) just as easily as loading data from a web page

Contributors: Albert Meroño, Rinke Hoekstra, Carlos Martínez

Copyright: Albert Meroño, Rinke Hoekstra, Carlos Martínez
License: MIT License (see LICENSE.txt)

Academic publications

  • Albert Meroño-Peñuela, Rinke Hoekstra. “grlc Makes GitHub Taste Like Linked Data APIs”. The Semantic Web – ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 – June 2, 2016, Revised Selected Papers. LNCS 9989, pp. 342-353 (2016). (PDF)
  • Albert Meroño-Peñuela, Rinke Hoekstra. “SPARQL2Git: Transparent SPARQL and Linked Data API Curation via Git”. In: Proceedings of the 14th Extended Semantic Web Conference (ESWC 2017), Poster and Demo Track. Portoroz, Slovenia, May 28th – June 1st, 2017 (2017). (PDF)
  • Albert Meroño-Peñuela, Rinke Hoekstra. “Automatic Query-centric API for Routine Access to Linked Data”. In: The Semantic Web – ISWC 2017, 16th International Semantic Web Conference. Lecture Notes in Computer Science, vol 10587, pp. 334-339 (2017). (PDF)
  • Pasquale Lisena, Albert Meroño-Peñuela, Tobias Kuhn, Raphaël Troncy. “Easy Web API Development with SPARQL Transformer”. In: The Semantic Web – ISWC 2019, 18th International Semantic Web Conference. Lecture Notes in Computer Science, vol 11779, pp. 454-470 (2019). (PDF)