Improve readme and documentation (#45)

* restructure readme * restructure mkdocs * add contributing doc * please linter * add readme for demo notebooks * add code of conduct * add readme for developer * add an empty changelog * fix link to test recipe * Apply suggestions from code review Co-authored-by: Bart Schilperoort <[email protected]> * address more comments from Bart * adress more comments from Bart * fix mkdocs content tabs --------- Co-authored-by: Bart Schilperoort <[email protected]>
EcoExtreML · Dec 11, 2023 · 7ad91da · 7ad91da
1 parent 24781ea
commit 7ad91da
Show file tree

Hide file tree

Showing 20 changed files with 518 additions and 127 deletions.
diff --git a/CITATION.cff b/CITATION.cff
@@ -0,0 +1,30 @@
+# YAML 1.2
+---
+cff-version: "1.1.0"
+title: "zampy"
+authors:
+  -
+    affiliation: "Netherlands eScience Center"
+    family-names: Schilperoort
+    given-names: Bart
+    orcid: "https://orcid.org/0000-0003-4487-9822"
+  -
+    affiliation: "Netherlands eScience Center"
+    family-names: Alidoost
+    given-names: Sarah
+    orcid: "https://orcid.org/0000-0001-8407-6472"
+  -
+    affiliation: "Netherlands eScience Center"
+    family-names: Liu
+    given-names: Yang
+    orcid: "https://orcid.org/0000-0002-1966-8460"
+
+date-released: <add release date>
+doi: <insert your DOI here>
+version: "0.0.1"
+repository-code: "https://github.com/EcoExtreML/zampy"
+keywords:
+  - data preparation
+  - land surface modelling
+message: "If you use this software, please cite it using these metadata."
+license: Apache-2.0
diff --git a/README.md b/README.md
@@ -1,64 +1,112 @@
 # zampy
-Tool for downloading Land Surface Model input data
+Tool for downloading Land Surface Model (LSM) input data.
+
+Named after *Zam*; [the Avestan language term for the Zoroastrian concept of "earth"](https://en.wikipedia.org/wiki/Zam).
 
 [![github license badge](https://img.shields.io/github/license/EcoExtreML/zampy)](https://github.com/EcoExtreML/zampy)
+[![Documentation Status](https://readthedocs.org/projects/zampy/badge/?version=latest)](https://zampy.readthedocs.io/en/latest/?badge=latest)
 [![build](https://github.com/EcoExtreML/zampy/actions/workflows/build.yml/badge.svg)](https://github.com/EcoExtreML/zampy/actions/workflows/build.yml)
 [![workflow scc badge](https://sonarcloud.io/api/project_badges/measure?project=EcoExtreML_zampy&metric=coverage)](https://sonarcloud.io/dashboard?id=EcoExtreML_zampy)
 
 
-## Tool outline:
+## Outline
+`zampy` is designed to retrieve data for LSM model input. It can help you prepare the data within the following steps:
+1. **Download** the data for the specified location(s) / geographical area.
+2. **Ingest** data into unified (zampy) format.
+3. **Load** the variables in a standardized way (standardized names & standardized units).
+4. **Convert** the data to standard formats:
+    - ALMA / PLUMBER2's ALMA formatted netCDF.
+    - *CMOR formatted netCDF*.
+
+(Note: items in *italic* will not be worked on for now/low priority, but we want to allow space for these in the future.)
+
+## Getting start
+
+### Installation
+[![workflow pypi badge](https://img.shields.io/pypi/v/zampy.svg?colorB=blue)](https://pypi.python.org/project/zampy/)
+[![supported python versions](https://img.shields.io/pypi/pyversions/zampy)](https://pypi.python.org/project/zampy/)
+
+To install the latest release of `zampy`, do:
+```console
+python3 -m pip install zampy
+```
+
+To install the in-development version from the GitHub repository, do:
 
- - Goal is to retrieve data for LSM model input.
-    1. First **download** the data for the specified location(s) / geographical area.
-    2. Be able to **load** the variables in a standardized way (standardized names & standardized units).
-    3. **Output** the data to standard formats:
-       - ALMA / PLUMBER2's ALMA formatted netCDF.
-       - *CMOR formatted netCDF*.
- - User-interaction should go through recipes. For example, see [springtime](https://github.com/phenology/springtime/blob/main/tests/recipes/daymet.yaml).
-   - Recipes define:
-     - data folder (where data should be downloaded to)
-     - time extent.
-     - spatial location / bounding box.
-     - datasets to be used
-       - variables within datasets
-   - Load recipes using Pydantic ([for example](https://github.com/phenology/springtime/blob/main/src/springtime/datasets/daymet.py)).
- - Support both a CLI & Python API.
+```console
+python3 -m pip install git+https://github.com/EcoExtreML/zampy.git
+```
 
-Note: items in *italic* will not be worked on for now/low priority, but we want to allow space for these in the future.
+### Configuration
+`Zampy` needs to be configured with a simple configuration file.
 
-## Instructions for CDS datasets (e.g. ERA5)
-To download the following datasets, users need access to CDS via cdsapi:
+You need to create this file under your user home directory:
 
-- ERA5
-- ERA5 land
-- LAI
-- land cover
+`~/.config/zampy/zampy_config.yml`
 
-First, you need to be a registered user on *CDS* via the [registration page](https://cds.climate.copernicus.eu/user/register?destination=%2F%23!%2Fhome).
+The configuration file should contain the `working_directory`, for instance:
+```yaml
+working_directory: /path_to_a_working_directory/  #for example: /home/bart/Zampy
+```
 
-Before submitting any request with `zampy`, please configure your zampy config file `zampy_config.yml` and put your `cdsapi` credentials in `zampy_config.yml`. Here is a short [instruction](https://cds.climate.copernicus.eu/api-how-to) about how to find your CDS API key. You can skip the steps related to `.cdsapirc` and instead create `zampy_config.yml` file under your _*user home*_/.config directory i.e. `~/.config/zampy/zampy_config.yml`. Then, add the key as: 
+If you need access to data on CDS or ADS server, you should add your CDS or ADS credentials to `zampy_config.yml`:
 
 ```yaml
 cdsapi:
   url:  # for example https://cds.climate.copernicus.eu/api/v2
   key:  # for example 12345:xhashd-232jcsha-dsaj429-cdjajd29319
+adsapi:
+  url:  # for example https://ads.atmosphere.copernicus.eu/api/v2
+  key:  # for example 12345:xhashd-232jcsha-dsaj429-cdjajd29319
 ```
 
-## Instructions for ADS datasets (e.g. CAMS EGG4)
-To download the following datasets, users need access to ADS via cdsapi:
+About how to create CDS or ADS credentials, check the section below.
 
-- CAMS EGG4 (e.g. co2)
+### How to use `zampy`
 
-First, you need to be a registered user on *ADS* via the [registration page](https://ads.atmosphere.copernicus.eu/user/register?destination=%2F%23!%2Fhome).
+We recommend our users to use `zampy` with recipes.
 
-Before submitting any request with `zampy`, please configure your zampy config file `zampy_config.yml` and put your `cdsapi` credentials in `zampy_config.yml`. Here is a short [instruction](https://cds.climate.copernicus.eu/api-how-to) about how to find your CDS API key. You can skip the steps related to `.cdsapirc` and instead create `zampy_config.yml` file under your _*user home*_/.config directory i.e. `~/.config/zampy/zampy_config.yml`. Then, add the key as: 
+A "recipe" is a file with yml extension, it defines:
+- data downloading
+  - time extent.
+  - spatial location / bounding box.
+  - datasets to be downloaded
+    - variables within datasets
+- data conversion
+  - convert to desired [conventions](./src/zampy/conventions/)
+  - output frequency
+  - output resolution
 
-```yaml
-adsapi:
-  url:  # for example https://ads.atmosphere.copernicus.eu/api/v2
-  key:  # for example 12345:xhashd-232jcsha-dsaj429-cdjajd29319
+A sample recipe can be found in the [documentation](https://zampy.readthedocs.io/en/latest/#how-to-use-zampy).
+
+When you have your reciped created and saved on your disk, you can execute your recipe by running the following code in your shell:
+
+```py
+zampy /path_to_recipe/sample_recipe.yml
 ```
 
+We also provide python API for you to intereact with `zampy`. You can find the example notebooks for each supported dataset [here](./demo/).
+
+## Instructions for CDS/ADS datasets
+
+To download the following datasets, users need access to CDS/ADS via `cdsapi`/`adsapi`:
+- CDS
+  - ERA5
+  - ERA5 land
+  - LAI
+  - land cover
+- ADS
+  - CAMS EGG4 (e.g. co2)
+
+To generate these API keys, you need to be a registered user on *CDS* via the [registration page](https://cds.climate.copernicus.eu/user/register?destination=%2F%23!%2Fhome), or on *ADS* via the [registration page](https://ads.atmosphere.copernicus.eu/user/register?destination=%2F%23!%2Fhome).
+
+Before submitting any request with `zampy`, please put your `cdsapi`/`adsapi` credentials in `zampy_config.yml`. Here is a short [instruction](https://cds.climate.copernicus.eu/api-how-to) about how to find your CDS/ADS API key. You can skip the steps related to `.cdsapirc` and simply add the key to `zampy_config.yml`.
+
 ### Agree to the Terms of Use on CDS/ADS
 
-When downloading a dataset for the first time, it is **necessary to agree to the Terms of Use of every datasets that you intend to download**. This can only be done via the CDS website. When you try to download these datasets, you will be prompted to go to the terms of use and accept them.
+When downloading a dataset for the first time, it is **necessary to agree to the Terms of Use of every datasets that you intend to download**. This can only be done via the CDS/ADS website. When you try to download these datasets, you will be prompted to go to the terms of use and accept them.
+
+
+## Acknowledgements
+
+This package was developed by the Netherlands eScience Center. Development was supported by the Netherlands eScience Center under grant number NLESC.ASDI.2020.026.
diff --git a/demo/README.md b/demo/README.md
@@ -0,0 +1,11 @@
+# Using `zampy` in notebooks
+
+It is possible to intereact with `zampy` via Python APIs. Here are some demo notebooks showing how to use `zampy` inside a Jupyter notebook environment for each supported dataset:
+
+ - [ERA5](./era5_dataset_demo.ipynb)
+ - [ERA5-land](./era5-land_dataset_demo.ipynb)
+ - [ETH canopy height](./eth_dataset_demo.ipynb)
+ - [PRISM DEM](./prism_dem_demo.ipynb)
+ - [CAMS EGG4](./cams_co2_dataset_demo.ipynb)
+ - [Land cover classification gridded maps](./land_cover_dataset_demo.ipynb)
+
diff --git a/demo/cams_co2_dataset_demo.ipynb b/demo/cams_co2_dataset_demo.ipynb
@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "### Handle CAMS EGG4 dataset with Zampy\n",
-    "Demo notebook for developers."
+    "Demo notebook for users and developers."
    ]
   },
   {
@@ -35,7 +35,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "work_dir = Path(\"/home/yangliu/EcoExtreML/temp\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2003-01-02\"), np.datetime64(\"2003-01-31\"))\n",

diff --git a/demo/era5-land_dataset_demo.ipynb b/demo/era5-land_dataset_demo.ipynb
@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "### Handle ERA5 land dataset with Zampy\n",
-    "Demo notebook for developers."
+    "Demo notebook for users and developers."
    ]
   },
   {
@@ -35,7 +35,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "work_dir = Path(\"/home/yangliu/EcoExtreML/temp\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2010-01-01T00:00:00\"), np.datetime64(\"2010-01-31T23:00:00\"))\n",

diff --git a/demo/era5_dataset_demo.ipynb b/demo/era5_dataset_demo.ipynb
@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "### Handle ERA5 dataset with Zampy\n",
-    "Demo notebook for developers."
+    "Demo notebook for users and developers."
    ]
   },
   {
@@ -35,7 +35,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "work_dir = Path(\"/home/yangliu/EcoExtreML/temp\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2010-01-01T00:00:00\"), np.datetime64(\"2010-01-31T23:00:00\"))\n",

diff --git a/demo/eth_dataset_demo.ipynb b/demo/eth_dataset_demo.ipynb
@@ -6,7 +6,7 @@
    "metadata": {},
    "source": [
     "### Handle ETH Canopy Height dataset with Zampy\n",
-    "Demo notebook for developers."
+    "Demo notebook for users and developers."
    ]
   },
   {
@@ -28,7 +28,7 @@
     "from zampy.datasets.dataset_protocol import TimeBounds, SpatialBounds\n",
     "from pathlib import Path\n",
     "\n",
-    "work_dir = Path(\"/home/bart/Zampy\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2020-01-01\"), np.datetime64(\"2020-12-31\"))\n",

diff --git a/demo/land_cover_dataset_demo.ipynb b/demo/land_cover_dataset_demo.ipynb
@@ -35,7 +35,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "work_dir = Path(\"/home/yangliu/EcoExtreML/temp\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2010-01-01T00:00:00\"), np.datetime64(\"2011-01-31T23:00:00\"))\n",

diff --git a/demo/prism_dem_demo.ipynb b/demo/prism_dem_demo.ipynb
@@ -6,15 +6,11 @@
    "metadata": {},
    "source": [
     "### Download the Prism DEM dataset with Zampy\n",
-    "Demo notebook for developers.\n",
+    "Demo notebook for users and developers.\n",
     "\n",
-    "\n",
-    "Note:\n",
-    "```\n",
-    "    two versions of this dataset are available, a 30 m and a 90 m resolution one.\n",
-    "    These are considered to be different datasets, and can be downloaded by using\n",
-    "    either the PrismDEM30 or the PrismDEM90 class.\n",
-    "```"
+    "Note: <br>\n",
+    "Two versions of this dataset are available, a 30 m and a 90 m resolution one. <br>\n",
+    "These are considered to be different datasets, and can be downloaded by using either the PrismDEM30 or the PrismDEM90 class."
    ]
   },
   {
@@ -36,7 +32,7 @@
     "from zampy.datasets.dataset_protocol import TimeBounds, SpatialBounds\n",
     "from pathlib import Path\n",
     "\n",
-    "work_dir = Path(\"/home/bart/Zampy\")\n",
+    "work_dir = Path(\"/path_to_work_directory\")\n",
     "download_dir = work_dir / \"download\"\n",
     "ingest_dir = work_dir / \"ingest\"\n",
     "times = TimeBounds(np.datetime64(\"2020-01-01\"), np.datetime64(\"2020-12-31\"))\n",
@@ -653,7 +649,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.0"
   },
   "orig_nbformat": 4
  },

diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -0,0 +1,13 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added:
+
+### Removed:
+
+### Changed: