Work on jupyter (#23)

* minor text changes * Unfinished debugging * Almost ready, need to re-run with complete forcing * Add shapefile to repo * Run case0 notebook on Jupyter * Run case1 on Jupyter machine with pre-installed forcing and parameters * Add lisflood maskmap to repo * Confirm case2 can run on Jupyter. Didn't complete a full run, output is "old" * Update and fully run case 3 * Not sure what I changed since last commit * Make case4 notebook runnable on jupyter * Add figures to repo Co-authored-by: Rolf Hut <[email protected]>
eWaterCycle · Aug 27, 2021 · 264fe75 · 264fe75
1 parent 728e81b
commit 264fe75
Show file tree

Hide file tree

Showing 18 changed files with 2,736 additions and 2,859 deletions.
diff --git a/Case-1_Marrmot_Merrimack_ESMValTool.ipynb b/Case-1_Marrmot_Merrimack_ESMValTool.ipynb
diff --git a/Case0_Marrmot_Merrimack_ESMValTool.ipynb b/Case0_Marrmot_Merrimack_ESMValTool.ipynb
diff --git a/Case1CoupleMarrmotAndPCRGlobWB.ipynb → Case2CoupleMarrmotAndPCRGlobWB.ipynb b/Case1CoupleMarrmotAndPCRGlobWB.ipynb → Case2CoupleMarrmotAndPCRGlobWB.ipynb
diff --git a/Case2ForcePCRGlobWBWithadditionalEvapInfo.ipynb b/Case2ForcePCRGlobWBWithadditionalEvapInfo.ipynb
diff --git a/Case3ForcePCRGlobWBWithadditionalEvapInfo.ipynb b/Case3ForcePCRGlobWBWithadditionalEvapInfo.ipynb
diff --git a/Case3_ForcePCRGlobWithAdditionalDataHydrograph.png b/Case3_ForcePCRGlobWithAdditionalDataHydrograph.png
diff --git a/Case3_ForcePCRGlobWithAdditionalDataMap.png b/Case3_ForcePCRGlobWithAdditionalDataMap.png
diff --git a/case0HelloWorldHydrograph.png b/case0HelloWorldHydrograph.png
diff --git a/case0_wflow_LISFlood_merrimack_inc_ESMValtool.ipynb b/case0_wflow_LISFlood_merrimack_inc_ESMValtool.ipynb
diff --git a/case1HelloWorldsHydrograph.png b/case1HelloWorldsHydrograph.png
diff --git a/case1_wflow_LISFlood_merrimack_inc_ESMValtool.ipynb b/case1_wflow_LISFlood_merrimack_inc_ESMValtool.ipynb
diff --git a/...MOT_calibration_ensemble_RunSavanah.ipynb → ...MOT_calibration_ensemble_RunSavanah.ipynb b/...MOT_calibration_ensemble_RunSavanah.ipynb → ...MOT_calibration_ensemble_RunSavanah.ipynb
@@ -2,49 +2,61 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "054c676c-b742-4fa0-93a0-539b03c6d133",
+   "id": "c5a90583",
    "metadata": {},
    "source": [
     "![ewatercycle logo](https://raw.githubusercontent.com/eWaterCycle/ewatercycle/main/docs/examples/logo.png)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "97582468-9dc3-4e7f-8e10-1a3afc869b53",
+   "id": "dffd5e98",
    "metadata": {},
    "source": [
-    "# Case study 3: Calibrate Marrmot M14 using GRDC data"
+    "# Case study 4: Calibrate Marrmot M14 using GRDC data"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a4b07a2b-5e50-422e-86aa-8f89e89679d4",
+   "id": "f7c740e1",
    "metadata": {},
    "source": [
     "This notebook demostractes how to use a model in eWatercycle, and calibrate it on a catchment (Savannah) and forcing data (ERA5). \n",
     "The model used in this notebook is: [MARRMoT](https://github.com/wknoben/MARRMoT) M14 (TOPMODEL)\n",
-    "We use [CMA Evolution Strategy (CMA-ES) algorithm](https://github.com/CMA-ES/pycma) package for finding the best parameters for the model. The procedure is explained in the cells below."
+    "We use [CMA Evolution Strategy (CMA-ES) algorithm](https://github.com/CMA-ES/pycma) package for finding the best parameters for the model. The procedure is explained in the cells below.\n",
+    "\n",
+    "\n",
+    "***NOTE: this notebooks is computationally expensive. Although it might possible to execute it on the demo machine (be sure to adapt the POPSIZE and MAXITER settings below), this is not recommended. This notebook has been executed on a machine with 24 cpu cores and took about 4 hours to complete. It is included here to demonstrate what is possible with ewatercycle, not as a performance benchmark. Please use it with care.***"
    ]
   },
   {
-   "cell_type": "markdown",
-   "id": "challenging-classroom",
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "21bb49ea",
    "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Collecting cma\n",
+      "  Downloading cma-3.1.0-py2.py3-none-any.whl (269 kB)\n",
+      "\u001b[K     |████████████████████████████████| 269 kB 12 kB/s s eta 0:00:01\n",
+      "\u001b[?25hInstalling collected packages: cma\n",
+      "Successfully installed cma-3.1.0\n"
+     ]
+    }
+   ],
    "source": [
-    "Make sure your ewatercycle conda environment is [activated](https://github.com/eWaterCycle/ewatercycle#install)."
+    "# Install the CMA package if not available yet\n",
+    "!pip install --user cma"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "828160f0-db60-4073-b836-fa2d8952f804",
+   "execution_count": null,
+   "id": "2c8216fe",
    "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:27.278577Z",
-     "iopub.status.busy": "2021-08-20T07:11:27.277934Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.877165Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.877684Z"
-    },
     "tags": []
    },
    "outputs": [],
@@ -77,7 +89,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8cd9aab-8416-44a7-969e-367b092995d1",
+   "id": "bb71fd2a",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-09T15:05:52.535716Z",
@@ -95,20 +107,12 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "651bac1f-728b-48b1-a9ab-306164756402",
+   "id": "9bebaff3",
    "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:45.883168Z",
-     "iopub.status.busy": "2021-08-20T07:11:45.882575Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.884604Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.885095Z"
-    },
     "tags": []
    },
    "outputs": [],
    "source": [
-    "SHAPE_DIR = Path(\"/projects/0/wtrcycle/comparison/recipes_auxiliary_datasets/Lorentz_Basin_Shapefiles\")\n",
-    "\n",
     "# time range for calibration\n",
     "CALIBRATION = {\n",
     "    'spinup_start': '1990-01-01T00:00:00Z',\n",
@@ -119,7 +123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "intensive-marriage",
+   "id": "edfdc84c",
    "metadata": {},
    "source": [
     "## 1. Loading forcing data"
@@ -128,62 +132,40 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "experienced-consumption",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:45.889131Z",
-     "iopub.status.busy": "2021-08-20T07:11:45.888540Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.902396Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.902919Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "FORCING = load_foreign('marrmot',\n",
-    "                        directory='/projects/0/wtrcycle/comparison/forcing/marrmot-m01',\n",
-    "                        start_time=CALIBRATION['spinup_start'],\n",
-    "                        end_time=CALIBRATION['end'],\n",
-    "                        forcing_info={\n",
-    "                            'forcing_file': 'marrmot-m01_ERA5_Savannah_1990_2001.mat'\n",
-    "                        })"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "expired-oxford",
+   "id": "863ad1b9",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "## 2. Loading eWaterCycle configuration file\n",
-    "\n",
-    "Config settings on Cartesius:\n",
-    "``` yaml\n",
-    "grdc_location: '/projects/0/wtrcycle/comparison/GRDC/GRDC_GCOSGTN-H_27_03_2019'\n",
-    "container_engine: 'singularity'\n",
-    "singularity_dir: '/scratch/shared/ewatercycle/case3_marrmot14'\n",
-    "output_dir: '/scratch/shared/ewatercycle/case3_marrmot14'\n",
-    "```"
+    "FORCING = ewatercycle.forcing.load(\"/mnt/data/forcing/marrmot-m01_ERA5_1990-2001_savannah\")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "wooden-walnut",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:45.907010Z",
-     "iopub.status.busy": "2021-08-20T07:11:45.906199Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.910968Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.910439Z"
+   "id": "e8f244c9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "eWaterCycle forcing\n",
+      "-------------------\n",
+      "start_time=1990-01-01T00:00:00Z\n",
+      "end_time=2001-12-31T00:00:00Z\n",
+      "shape=/mnt/data/forcing/marrmot-m01_ERA5_1990-2001_savannah/Savannah.shp\n",
+      "forcing_file=marrmot-m01_ERA5_Savannah_1990_2001.mat\n",
+      "directory=/mnt/data/forcing/marrmot-m01_ERA5_1990-2001_savannah\n"
+     ]
     }
-   },
-   "outputs": [],
+   ],
    "source": [
-    "CFG.load_from_file('./ewatercycle.yaml')"
+    "print(FORCING)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c99bfe61-b501-4826-a130-f59836b9d3c6",
+   "id": "42896938",
    "metadata": {},
    "source": [
     "## 3. Functions for calibration\n",
@@ -220,7 +202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e121ee14-a857-47cd-a117-dc6c4256aa70",
+   "id": "8987985c",
    "metadata": {},
    "source": [
     "Here, we can set `popsize` and `maxiter` in the cma-es options:"
@@ -229,15 +211,8 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "94a695a3-924b-4bb0-befb-a673f6be15cc",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:45.914767Z",
-     "iopub.status.busy": "2021-08-20T07:11:45.914190Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.916062Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.916529Z"
-    }
-   },
+   "id": "fec6e64d",
+   "metadata": {},
    "outputs": [],
    "source": [
     "POPSIZE = 18  # it can be equal to number of available cores/2\n",
@@ -247,15 +222,8 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "separate-unemployment",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2021-08-20T07:11:45.932101Z",
-     "iopub.status.busy": "2021-08-20T07:11:45.918970Z",
-     "iopub.status.idle": "2021-08-20T07:11:45.934046Z",
-     "shell.execute_reply": "2021-08-20T07:11:45.934513Z"
-    }
-   },
+   "id": "feda0536",
+   "metadata": {},
    "outputs": [],
    "source": [
     "def run_model(parameters):\n",
@@ -304,7 +272,7 @@
     "\n",
     "def get_catchment_area(catchment):\n",
     "    \"\"\"Get the catchment area in m2.\"\"\"\n",
-    "    shapefile = SHAPE_DIR / catchment / f'{catchment}.shp'\n",
+    "    shapefile = FORCING.shape\n",
     "    shape = shapereader.Reader(shapefile)\n",
     "    attr = next(shape.records())\n",
     "    return attr.attributes['SUB_AREA'] * 1e6\n",
@@ -416,7 +384,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "lined-python",
+   "id": "7270f93d",
    "metadata": {},
    "source": [
     "## 4. Running the calibration"
@@ -425,7 +393,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "indie-separate",
+   "id": "e19135a5",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-20T07:11:45.939046Z",
@@ -470,7 +438,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a0ece587-8147-49ad-ac1b-a3772ec7f06d",
+   "id": "ca0b056f",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-09T18:35:28.457881Z",
@@ -486,7 +454,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "22dca4ac-e0ce-4963-802d-9b3d892c4da6",
+   "id": "e19fbdeb",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-20T11:06:46.726584Z",
@@ -529,7 +497,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "hourly-rachel",
+   "id": "2a60732b",
    "metadata": {},
    "source": [
     "## 5. Plotting the optimization results"
@@ -538,7 +506,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "physical-wyoming",
+   "id": "8d96f4b3",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-20T11:06:46.773710Z",
@@ -616,7 +584,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "d1f73a0d-95df-4836-8915-0cc9eb150602",
+   "id": "ebf65971",
    "metadata": {
     "execution": {
      "iopub.execute_input": "2021-08-20T11:06:46.819482Z",
@@ -652,15 +620,15 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "royal-living",
+   "id": "41172288",
    "metadata": {},
    "outputs": [],
    "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
@@ -674,7 +642,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.2"
+   "version": "3.7.7"
   }
  },
  "nbformat": 4,

diff --git a/settingFiles/Merrimack.dbf b/settingFiles/Merrimack.dbf
diff --git a/settingFiles/Merrimack.prj b/settingFiles/Merrimack.prj
@@ -0,0 +1 @@
+GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]]
diff --git a/settingFiles/Merrimack.qpj b/settingFiles/Merrimack.qpj
@@ -0,0 +1 @@
+GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]]
diff --git a/settingFiles/Merrimack.shp b/settingFiles/Merrimack.shp
diff --git a/settingFiles/Merrimack.shx b/settingFiles/Merrimack.shx
diff --git a/settingFiles/lisflood_model_mask.nc b/settingFiles/lisflood_model_mask.nc