docs/notebooks/run_model_with_different_dataset.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running the STEMMUS_SCOPE model with different dataset\n",
    "\n",
    "This notebook shows how to run the STEMMUS_SCOPE model uisng the `Executable` file, including preprocessing and postprocessing. For postprocesisng, the file `required_netcdf_variables.csv` can be found at https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/utils/csv_to_nc/required_netcdf_variables.csv.\n",
    "\n",
    "See the [Running the STEMMUS_SCOPE model on a system](./run_model_on_different_infra.ipynb) for different ways e.g. Executable file, Matlab, Octave.\n",
    "The `STEMMUS_SCOPE` executable file is located in a repository https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/exe/STEMMUS_SCOPE.\n",
    "\n",
    "## Which datasets are available?\n",
    "The config file shows how to set the paths to different data needed by the model. Check the data source on Snelius and CRIB and modify the config file accordingly:\n",
    "\n",
    "#### On CRIB\n",
    "- Site data:\n",
    "    - path to data: `/data/shared/EcoExtreML/STEMMUS_SCOPEv1.0.0/input`,\n",
    "    - config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/config_file_crib.txt\n",
    "- Global data:\n",
    "    - path to data `/data/shared/EcoExtreML/STEMMUS_SCOPEv1.0.0/global_data`,\n",
    "    - config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_crib_global.txt\n",
    "\n",
    "#### On Snellius\n",
    "- Site data:\n",
    "    - path to data: `/projects/0/ttse0619/`,\n",
    "    - config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_snellius.txt\n",
    "- Global data:\n",
    "    - path to data: `/projects/0/ttse0619/global_data`,\n",
    "    - config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_snellius_global.txt\n",
    "\n",
    "Choose how do you want to run the model:\n",
    "\n",
    "[**Using site data**](#using-site-data)\n",
    "\n",
    "[**Using global data**](#using-global-data)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using site data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from PyStemmusScope import StemmusScope\n",
    "from PyStemmusScope import save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user must provide the correct path\n",
    "path_to_config_file = \"./my_config_template.txt\"\n",
    "path_to_exe_file = \"./STEMMUS_SCOPE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run the model on your own machine, make sure `LD_LIBRARY_PATH` is set correctly. To do this, uncomment the cell below and run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/MATLAB/MATLAB_Runtime/R2023a/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/extern/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64\n"
     ]
    }
   ],
   "source": [
    "# # Set LD_LIBRARY_PATH\n",
    "# matlab_path = !whereis MATLAB\n",
    "# matlab_path = matlab_path.s.split(\": \")[1]\n",
    "# os.environ['LD_LIBRARY_PATH'] = (\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64\")\n",
    "# print(os.environ['LD_LIBRARY_PATH'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an an instance of the model\n",
    "model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New config file /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_config.txt\n",
      "Model input dir /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/\n",
      "Model output dir /home/sarah/temp/ecoextreml/test/output/ZA-Kru_2024-01-24-1125/\n"
     ]
    }
   ],
   "source": [
    "# setup the model\n",
    "# here you can change the location and start & end time\n",
    "config_path = model.setup(\n",
    "    Location=\"ZA-Kru\",\n",
    "    StartTime=\"2001-01-01T00:00\",\n",
    "    EndTime=\"2001-01-02T01:30\",\n",
    ")\n",
    "\n",
    "# new config file genertaed to run the model \n",
    "print(f\"New config file {config_path}\")\n",
    "\n",
    "# see input and output paths generated by the model\n",
    "print(f'Model input dir {model.config[\"InputPath\"]}')\n",
    "print(f'Model output dir {model.config[\"OutputPath\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Reading config from /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_config.txt\\nFinished model initialization\\nThe calculations start now\\nFinalizing STEMMUS_SCOPE\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run the model\n",
    "result = model.run()\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/sarah/temp/ecoextreml/test/output/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_STEMMUS_SCOPE.nc\n"
     ]
    }
   ],
   "source": [
    "# save output in netcdf format\n",
    "required_netcdf_variables = \"./required_netcdf_variables.csv\"\n",
    "nc_file_name = save.to_netcdf(config_path, required_netcdf_variables)\n",
    "print(nc_file_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using global data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from PyStemmusScope import StemmusScope\n",
    "from PyStemmusScope import save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user must provide the correct path\n",
    "path_to_config_file = \"./my_config_template_global.txt\"\n",
    "path_to_exe_file = \"./STEMMUS_SCOPE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run the model on your own machine, make sure `LD_LIBRARY_PATH` is set correctly. To do this, uncomment the cell below and run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/MATLAB/MATLAB_Runtime/R2023a/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/extern/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64\n"
     ]
    }
   ],
   "source": [
    "# # Set LD_LIBRARY_PATH\n",
    "# matlab_path = !whereis MATLAB\n",
    "# matlab_path = matlab_path.s.split(\": \")[1]\n",
    "# os.environ['LD_LIBRARY_PATH'] = (\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:\"\n",
    "#     f\"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64\")\n",
    "# print(os.environ['LD_LIBRARY_PATH'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an an instance of the model\n",
    "model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16\n",
      "  result = blockwise(\n",
      "/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16\n",
      "  result = blockwise(\n",
      "/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16\n",
      "  result = blockwise(\n",
      "/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16\n",
      "  result = blockwise(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New config file /home/sarah/temp/ecoextreml/test/input/global_N61-847_E24-295_2024-02-02-1620/global_2024-02-02-1620_config.txt\n",
      "Model input dir /home/sarah/temp/ecoextreml/test/input/global_N61-847_E24-295_2024-02-02-1620/\n",
      "Model output dir /home/sarah/temp/ecoextreml/test/output/global_N61-847_E24-295_2024-02-02-1620/\n"
     ]
    }
   ],
   "source": [
    "# setup the model\n",
    "config_path = model.setup(\n",
    "    Location=\"(61.8474, 24.2948)\",\n",
    "    StartTime=\"2014-01-10T00:00\",\n",
    "    EndTime=\"2014-01-15T00:00\",\n",
    ")\n",
    "\n",
    "# new config file genertaed to run the model \n",
    "print(f\"New config file {config_path}\")\n",
    "\n",
    "# see input and output paths generated by the model\n",
    "print(f'Model input dir {model.config[\"InputPath\"]}')\n",
    "print(f'Model output dir {model.config[\"OutputPath\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run the model\n",
    "result = model.run()\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save output in netcdf format\n",
    "required_netcdf_variables = \"./required_netcdf_variables.csv\"\n",
    "nc_file_name = save.to_netcdf(config_path, required_netcdf_variables)\n",
    "print(nc_file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "281eb038f5f4b6dc08cc25b19e3f510bd63483fa656fb888636d147f45c9c6d5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}