Catalog API Browser: Initial Strategy and Approach
From issue https://github.com/hotosm/OpenAerialMap/issues/5
The first version of OpenAerialMap Catalog will be a reference implementation of the Open Imagery Network (OIN). As a reference implementation, it will be a model for how the Network can be indexed and searched to provide access to openly licensed imagery.
Below is an out for defining the problem, understanding initial key user stories, and outlining a technical approach. This strategy will result in version 1.0 of the OAM Catalog.
The main challenge for the Catalog revolves around indexing the OIN on the backend and enabling fast searching on the frontend.
The initial development will focus on two key user stories:
- User who wants to trace for OSM (wants TMS endpoints for individual scenes or mosaics)
- Answers question: what can I do online now?
- HOT staff and volunteers looking to trace for OSM will be able to make use of the tool to find and use imagery available. Users would come to the site specifically looking to find a TMS endpoint to use within iD or JOSM. Some users may come directly from the HOT Tasking Manager and be looking to find new or updated imagery for a related task. Other users may be coming to the site to look for imager in an unmapped or relevant location.
- Experienced imagery user (wants raw imagery files)
- Answers question: what can I use to do more?
- Some HOT staff and volunteers may come to the site looking to use their experience to access imagery. Users will be looking to find, download, and then use the imagery locally. Users may know a specific location for where to search.
The catalog needs to parse and index AWS S3 buckets to generate a catalog of available imagery within the Network. Imagery within the network will have individual metadata, plus there will be metadata information for the provider of the S3 bucket. Below is a list of functions the catalog will need to perform:
- store provider’s S3 bucket information
- search and index S3 buckets
- parse JSON metadata files
- advance search for metadata
- add TMS information to metadata
- analytics
- simple authentication system for add/update mechanism
The catalog browser will need to provide a fast and intuitive interface for searching available imagery. By focusing on the two main use cases the main functions the browser will need to perform are:
- filter by date & time, imagery resolution, provider
- show available imagery on a map
- display imagery metadata (location, footprint?, date, time, resolution)
- provide access to TMS endpoints
- provide download functionality (needs to know file size as well)
Build v1.0 of an application that will provide an API for storing and searching OAM metadata. In production the API is backed by an Elasticsearch instance to improve performance. This first version of an OAM catalog will be the initial implementation of an application that will index the Open Imagery Network and provide a datastore and API access to frontend tools.
This application will focus on three main tasks:
- Index OIN master list
- Metadata data store
- Search API
Tech stack:
The API is built in Node using Hapi.js framework. The datastore is MongoDB and in production it is backed by Elasticsearch for better performance.
Technical diagram:
Build a v1.0 of an application that will provide search and access to the OAM catalog. This first version will be the initial implementation of an application for OAM. v1.0 will help establish a brand and UI look, as well as, a foundation for user actions and features to be built upon.
Version 1.0 will focus on two core user flows:
- Finding and searching for available imagery on the network that can be traced with OpenStreetMap tools
- Search to find imagery to download and use locally
Filtering and exposing imagery features will enable these user flows.
Tech stack:
Frontend implementation will focus on lightweight javascript-driven site. Site will be built to run on GitHub Pages to lower maintenance and server costs. We will build using Sass for CSS management and compilation, and Gulp for automated build and deployments. We will package and document to enable running the site locally.
The frontend will be a single-page web application. A javascript framework like Backbone.js will be used for structure and organization. Libraries like React will be investigated to see how they can be leveraged within the application.