recipes that require input from different MIP tables not supported by forcing.generate()

[RolfHut]

foricing.generate() makes the assumption that all the variables that the recipe needs are available in a unique set of information contained in a dict, for example:

cmip_dataset_observation = {
    "dataset": "MPI-ESM1-2-HR",
    "project": "CMIP6",
    "grid": "gn",
    "exp": "historical",
    "ensemble": "r3i1p1f1",
}

followed by

pcrglobwb_forcing = ewatercycle.forcing.sources["PCRGlobWBForcing"].generate(
    dataset=cmip_dataset_observation,
    start_time="1990-01-01T00:00:00Z",
    end_time="1992-12-31T00:00:00Z",
    shape="./data/Rhine/Rhine.shp",
    extract_region={
        "start_latitude": y0 - esmvaltool_padding,
        "end_latitude": y1 + esmvaltool_padding,
        "start_longitude": x0 - esmvaltool_padding,
        "end_longitude": x1 + esmvaltool_padding,
        },
    start_time_climatology="1990-01-01T00:00:00Z",
    end_time_climatology="1992-01-01T00:00:00Z",
)

When generating forcing for WFLOW we discovered that sometimes some variables are not stored in the same MIP table as others. (Case in point: orog).

Suggested fix (though I would not know how to implement) would be to allow a nested dict in the dataset specificy this for expert users, for example:

cmip_dataset_observation = {
    "dataset": "MPI-ESM1-2-HR",
    "project": "CMIP6",
    "mip": {"tas": "day", "pr": "day", "orog": "perm"},
    "grid": "gn",
    "exp": "historical",
    "ensemble": "r3i1p1f1",
}

This could be an option for any field in the dict. When a single string is provided this automatically the value for all the variables.

[BSchilperoort]

Yeah it would be nice to fix this so we can use variables from multiple mip tables. I think defining it as a dict (mapping variables to mips) could be a good way to structure in the input.
Luckily the RecipeBuilder class already should support this.

For wflow this would be the definition by the way:

 "mip": {"tas": "day", "pr": "day", "orog": "fx", "rsdt", "CFday"},

[RolfHut]

Will you make the change and PR? Or should I?

[Peter9192]

But the forcing object knows which vars it needs, right? By specifying vars explicitly, will the user then overwrite the forcing object internals?

[BSchilperoort]

Yeah actually for wflow it should work fine:

        partial.add_variables(["tas", "pr", "psl", "rsds"])
        .add_variable("orog", mip="fx", start_year=False, end_year=False)
        .add_variable("rsdt", mip="CFday")

However, the UserForcing classes currently do not support this yet.

This makes adding another generic forcing (like the Makkink forcing) more difficult, as it only allows variables from one MIP table.

[RolfHut]

doesn't this assume that all data sources have the same variables in the same MIP tables, ie. "orog" must always be in "fx". Is that universally true? If ERA5 is hosted through ESGF than orog is in the same table as all the other variables. Wouldn't allowing the use to specify not be better? (potentially as an overwrite?)

[RolfHut]

This is fixed now, right? If so: @BSchilperoort will you close this?

[BSchilperoort]

all data sources have the same variables in the same MIP tables, ie. "orog" must always be in "fx". Is that universally true?

Yes. That should always be true. Variables can be in multiple tables, but that is generally only when these are on different frequencies (hourly, daily, monthly).

This is fixed now, right? If so: @BSchilperoort will you close this?

Well it works well for the complicated recipes, but not for the newer custom forcings (like the Makkink forcing). I'll keep it open for that use case.