Forcing builder (#365)

* Make generic distributed esmvaltool recipe

* New API for forcing generator

* Use fluent builder pattern to construct esmvaltool recipes

* Added builder.add_variables(['pr'])

* Working GenericDistributedForcing.generate

* Move all esmvaltool specific stuff to own package

This make base package much cleaner

* Move builder test

* Fix mypy errors

* Use builder for LisfloodForcing

* Dont use name of Python built-in package as module name

Got
Traceback (most recent call last):
  File "/home/verhoes/git/eWaterCycle/ewatercycle/src/ewatercycle/esmvaltool/diagnostic/copy.py", line 7, in <module>
    from esmvaltool.diag_scripts.shared import (
  File "/home/verhoes/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/esmvaltool/diag_scripts/shared/__init__.py", line 2, in <module>
    from . import io, iris_helpers, names, plot
  File "/home/verhoes/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/esmvaltool/diag_scripts/shared/io.py", line 5, in <module>
    from pprint import pformat
  File "/home/verhoes/mambaforge/envs/ewatercycle/lib/python3.10/pprint.py", line 38, in <module>
    import dataclasses as _dataclasses
  File "/home/verhoes/mambaforge/envs/ewatercycle/lib/python3.10/dataclasses.py", line 3, in <module>
    import copy
  File "/home/verhoes/git/eWaterCycle/ewatercycle/src/ewatercycle/esmvaltool/diagnostic/copy.py", line 7, in <module>
    from esmvaltool.diag_scripts.shared import (
ImportError: cannot import name 'ProvenanceLogger' from partially initialized module 'esmvaltool.diag_scripts.shared' (most likely due to a circular import) (/home/verhoes/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/esmvaltool/diag_scripts/shared/__init__.py)

* Added GenericLumpedForcing, untested as dcache is curently down

* Use recipe builder for MarrmotForcing

* Use builder in WflowForcing

* Use builder for ForcingHype

* Make mip required in dataset

* More docs

* Make expected recipe string consistent

* Parse esmvaltool recipe output when non netcdf files are produced

+ more docs + fix tests

* Got LisfloodForcing.generate(lisvap=None) working

Had to make Dataset.mip optional otherwise tdps var with mip=day throws esmvaltool error

* Make MarrmotForcing.generate work

* Fix tests for wflow and pcrglobwb

* Add support for variable without temporal selection

* give right dataset to lisvap

* Correct expected orog var

* Make mypy happy

* Dont copy line number, >>> and ... from code blocks

* More tests, flake8 fixes and docs

* Tests for DefaultForcing + Remove discriminator from forcing classes

As you now have to specify which class you want to use to load a forcing.

* Replaced with tests/src/base/test_forcing.py

* Remove discriminator from forcing classes from tests

* Make flake8 happier

* Model validator should return otherwise object is set to None

* Add test for MarrmotForcing.to_xarray()

* Add generic forcing to user guide + custom forcing instructions +

prevent circular deps with generic forcings in entry points

* More docs

* Use Dataset object instead of dict in pre defined datasets

* Tested GenericDistributedForcing with non-era dataset

* Allow extra attributes in Dataset

'*' in forcing files was caused by version attribute in Dataset being silently forgotten.
By allowing extra attributes the generate file has version value instead of `*`.

* Tested CMIP6

* Add dataset.version to expected recipes

* Add changes to CHANGELOG + more docs + more todos

* Fix tests with updated model names

* Disable lisflood model tests as downloading parameter set too slow

* Implement PR suggestions

* Rename build_recipe() to build_<modelname>_recipe()

* Centralize esmvaltool config for download

* Rename esmvaltool.models to esmvaltool.schema

When we talk about models we mean hydrological models.

---------

Co-authored-by: Peter Kalverla <[email protected]>

CHANGELOG.md

@@ -9,6 +9,11 @@ Formatted as described on [https://keepachangelog.com](https://keepachangelog.co
 ### Added
 
 - Apptainer support ([#290](https://github.com/eWaterCycle/ewatercycle/issues/290))
+- Forcing ((#365)[https://github.com/eWaterCycle/ewatercycle/pull/365]):
+  - GenericDistributedForcing class
+  - GenericLumpedForcing class
+  - Generate from not just ERA5 or ERA-Interim dataset, but any ESMvalTool supported dataset
+- Testing helpers for plugins ((#365)[https://github.com/eWaterCycle/ewatercycle/pull/365])
 
 ### Changed
 
@@ -18,6 +23,9 @@ Formatted as described on [https://keepachangelog.com](https://keepachangelog.co
 - Functions of a model inside a container that return the same result each call are cached with [MemoizedBmi](https://grpc4bmi.readthedocs.io/en/latest/api/grpc4bmi.bmi_memoized.html#grpc4bmi.bmi_memoized.MemoizedBmi) ([#339](https://github.com/eWaterCycle/ewatercycle/pull/339))
 - Moved CaseConfig to src/utils.py
 - forcing.load_foreign has been superceded by using sources.model(...)
+- Forcing ((#365)[https://github.com/eWaterCycle/ewatercycle/pull/365]):
+  - Instead of modifying an existing recipe now builds a ESMValTool recipe from scratch using a fluent interface
+  - DefaultForcing has overridable class methods for each step of the forcing generation process (build_recipe, run_recipe, recipe_output_to_forcing_arguments).
 
 ### Deprecated
 

docs/adding_models.rst

@@ -21,6 +21,7 @@ There are roughly five steps to adding a model to eWaterCycle:
 3. :ref:`Make recipe`
 4. :ref:`Add to Python package`
 5. :ref:`Add to platform`
+6. :ref:`Custom forcing`
 
 If you want to add a new version of a model the procedure is roughly the
 same, but you can skip several steps. If you are already familiar with the
@@ -238,3 +239,84 @@ Adding a new version of a model involves the following code changes:
 * Create new release of Python package. Done by package maintainers
 
 .. _scripts: https://github.com/eWaterCycle/infra/tree/main/roles/prep_shared_data
+
+.. _Custom forcing:
+
+Custom forcing
+==============
+
+If your model can use generic forcing data
+(:py:class:`~ewatercycle.base.forcing.GenericDistributedForcing` or :py:class:`~ewatercycle.base.forcing.GenericLumpedForcing`), you can skip this section.
+
+If your model needs custom forcing data, you need to create your own forcing class.
+
+The forcing class should sub class :py:class:`~ewatercycle.base.forcing.DefaultForcing`.
+
+In the class you have to define attributes for the forcing files your model will need.
+
+To use a ESMValTool recipe you have to implement the :py:meth:`~ewatercycle.base.forcing.DefaultForcing._build_recipe` method.
+It should return a :py:class:`~ewatercycle.esmvaltool.models.Recipe` object which can be build using the
+:py:class:`~ewatercycle.esmvaltool.builder.RecipeBuilder` class.
+For example if your model only needs precipitation you can implement the method like this:
+
+.. code-block:: python
+
+  from ewatercycle.forcing import RecipeBuilder
+
+  ...
+
+  @classmethod
+  def _build_recipe(cls,
+      start_time: datetime,
+      end_time: datetime,
+      shape: Path,
+      dataset: Dataset | str | dict = "ERA5",
+  ):
+      return (
+        RecipeBuilder()
+        .start(start_time.year)
+        .end(end_time.year)
+        .shape(shape)
+        .dataset(dataset)
+        .add_variable("pr")
+        .build()
+      )
+
+If your ESMValTool recipe needs additional arguments you can add and document them by implementing the :py:meth:`~ewatercycle.base.forcing.DefaultForcing.generate` method like
+so
+
+.. code-block:: python
+
+    @classmethod
+    def generate(
+      cls,
+      <arguments of DefaultForcing>,
+      my_argument: str,
+    ):
+        """Generate forcing data for my model.
+
+        Args:
+            <arguments of DefaultForcing>
+            my_argument: My argument
+        """
+        return super().generate(
+            <arguments of DefaultForcing>,
+            my_argument=my_argument,
+        )
+
+
+The recipe output is mapped to the forcing class arguments with the :py:meth:`~ewatercycle.base.forcing.DefaultForcing._recipe_output_to_forcing_arguments` method.
+If you want to change the mapping you can override this method.
+
+If you do not want to use ESMValTool to generate recipes you can override the :py:meth:`~ewatercycle.base.forcing.DefaultForcing.generate` method.
+
+To list your forcing class in :py:const:`ewatercycle.forcing.sources` you have to register in the `ewatercycle.forcings` entry point group.
+It can then be imported with
+
+.. code-block:: python
+
+  from ewatercycle.forcings import sources
+
+  forcing = source['MyForcing'](
+    ...
+  )

docs/conf.py

@@ -13,17 +13,6 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import sys
-# from pathlib import Path
-
-# src = Path(__file__).parent / ".." / "src"
-# sys.path.insert(0, str(src.absolute()))
-
-
 # -- General configuration ------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
@@ -201,3 +190,6 @@
     "sklearn": ("https://scikit-learn.org/stable", None),
     "xarray": ("https://docs.xarray.dev/en/stable/", None),
 }
+
+# Dont copy line number, >>> and ... from code blocks
+copybutton_exclude = ".linenos, .gp"