Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the capability to assimilate the MADIS snow depth data from GTS #1836

Merged
merged 13 commits into from
Sep 13, 2023
Merged

Add the capability to assimilate the MADIS snow depth data from GTS #1836

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
3a723e8
Initialize the branch to do the madis DA.
jiaruidong2017 Aug 23, 2023
46577a1
Merge branch 'develop' into feature/madis_da
jiaruidong2017 Aug 28, 2023
32e24b4
Merge branch 'develop' into feature/madis_da
jiaruidong2017 Sep 4, 2023
7f44dd4
Remove some repetitions.
jiaruidong2017 Sep 9, 2023
0273248
Fix the python norm error.
jiaruidong2017 Sep 9, 2023
4d3e105
Simplify the code to generate bufr2ioda YAML files
jiaruidong2017 Sep 9, 2023
9bce693
Remove the obsolete BUFR2IODAYAML and BUFRADPSFCYAML.
jiaruidong2017 Sep 9, 2023
0017549
Merge branch 'develop' into feature/madis_da
jiaruidong2017 Sep 11, 2023
60567df
Address reviewer's comments.
jiaruidong2017 Sep 11, 2023
dc83591
Simplify the codes following the reviewer's suggestions.
jiaruidong2017 Sep 11, 2023
94a0ff8
Remove the unnecessary construction of gts_yaml.
jiaruidong2017 Sep 11, 2023
04f7ec5
Merge branch 'develop' into feature/madis_da
jiaruidong2017 Sep 13, 2023
73fcc1c
Update the GDASApp's commit hash.
jiaruidong2017 Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parm/config/gfs/config.landanl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ echo "BEGIN: config.landanl"
# Get task specific resources
. "${EXPDIR}/config.resources" landanl

obs_list_name=gdas_land_adpsfc_only.yaml
obs_list_name=gdas_land_gts_only.yaml
if [[ "${cyc}" = "18" ]]; then
obs_list_name=gdas_land_prototype.yaml
fi
Expand Down
3 changes: 2 additions & 1 deletion parm/config/gfs/config.preplandobs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ echo "BEGIN: config.preplandobs"

export GTS_OBS_LIST="${HOMEgfs}/sorc/gdas.cd/parm/land/prep/prep_gts.yaml"
export BUFR2IODAX="${HOMEgfs}/exec/bufr2ioda.x"
export BUFR2IODAYAML="${HOMEgfs}/sorc/gdas.cd/test/testinput/bufr_adpsfc_snow.yaml"
export BUFRADPSFCYAML="${HOMEgfs}/sorc/gdas.cd/test/testinput/bufr_adpsfc_snow.yaml"
export BUFRSNOCVRYAML="${HOMEgfs}/sorc/gdas.cd/test/testinput/bufr_snocvr.yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these yamls be in parm/land/ and not test/?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the GTS bufr2ioda yamls in GDASApp are saved at the test/testinput/ directory.

export FIMS_NML_TMPL="${HOMEgfs}/sorc/gdas.cd/parm/land/prep/fims.nml.j2"
export IMS_OBS_LIST="${HOMEgfs}/sorc/gdas.cd/parm/land/prep/prep_ims.yaml"
export CALCFIMSEXE="${HOMEgfs}/exec/calcfIMS.exe"
Expand Down
39 changes: 32 additions & 7 deletions ush/python/pygfs/task/land_analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,12 +87,17 @@ def prepare_GTS(self) -> None:
logger.info("Copying GTS obs for bufr2ioda.x")
FileHandler(prep_gts_config.gtsbufr).sync()

# generate bufr2ioda YAML file
bufr2ioda_yaml = os.path.join(self.runtime_config.DATA, "bufr_adpsfc_snow.yaml")
logger.info(f"Generate BUFR2IODA YAML file: {bufr2ioda_yaml}")
temp_yaml = parse_j2yaml(self.task_config.BUFR2IODAYAML, self.task_config)
save_as_yaml(temp_yaml, bufr2ioda_yaml)
logger.info(f"Wrote bufr2ioda YAML to: {bufr2ioda_yaml}")
# generate bufr2ioda YAML files
adpsfc_yaml = os.path.join(self.runtime_config.DATA, "bufr_adpsfc_snow.yaml")
logger.info(f"Generate BUFR2IODA YAML file: {adpsfc_yaml}")
temp_yaml = parse_j2yaml(self.task_config.BUFRADPSFCYAML, self.task_config)
save_as_yaml(temp_yaml, adpsfc_yaml)
logger.info(f"Wrote bufr2ioda YAML to: {adpsfc_yaml}")
snocvr_yaml = os.path.join(self.runtime_config.DATA, "bufr_snocvr.yaml")
logger.info(f"Generate BUFR2IODA YAML file: {snocvr_yaml}")
temp_yaml = parse_j2yaml(self.task_config.BUFRSNOCVRYAML, self.task_config)
save_as_yaml(temp_yaml, snocvr_yaml)
logger.info(f"Wrote bufr2ioda YAML to: {snocvr_yaml}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we may be able to make this a bit less hard-coded and yaml driven from the top.
Say for e.g. somewhere in GDASApp/parm/land has bufr2ioda.yaml that contains:

bufr2ioda:
     adpsfc: {{ HOMEgfs }}/sorc/gdas.cd/test/testinput/bufr_adpsfc_snow.yaml
     snocvr: {{ HOMEgfs }}/sorc/gdas.cd/test/testinput/bufr_snocvr.yaml

We could then loop over this list to parse the yamls, and call the bufr2ioda.x and verify the output is produced.

The code then does not know what it is processing and bufr files can be added/removed to the yaml.

Would this be acceptable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may be able to use land/prep/prep_gts.yaml as it already seems to have the appropriate information and it is likely better to keep them together.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aerorahul Follow your suggestions, I have made the changes for your review. Thanks.


logger.info("Link BUFR2IODAX into DATA/")
exe_src = self.task_config.BUFR2IODAX
Expand All @@ -105,14 +110,34 @@ def prepare_GTS(self) -> None:
if os.path.isfile(f"{os.path.join(localconf.DATA, output_file)}"):
rm_p(output_file)

# execute BUFR2IODAX to convert GTS bufr data into IODA format
# execute BUFR2IODAX to convert adpsfc bufr data into IODA format
yaml_file = f"bufr_adpsfc_snow.yaml"
if not os.path.isfile(f"{os.path.join(localconf.DATA, yaml_file)}"):
logger.exception(f"{yaml_file} not found")
raise FileNotFoundError(f"{os.path.join(localconf.DATA, yaml_file)}")
exe = Executable(self.task_config.BUFR2IODAX)
exe.add_default_arg(os.path.join(localconf.DATA, f"{yaml_file}"))

logger.info(f"Executing {exe}")
try:
exe()
except OSError:
raise OSError(f"Failed to execute {exe}")
except Exception:
raise WorkflowException(f"An error occured during execution of {exe}")

output_file = f"{localconf.OPREFIX}snocvr_snow.nc4"
if os.path.isfile(f"{os.path.join(localconf.DATA, output_file)}"):
rm_p(output_file)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if os.path.isfile(f"{os.path.join(localconf.DATA, output_file)}"):
rm_p(output_file)
rm_p(output_file)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above lines were removed.


# execute BUFR2IODAX to convert snocvr bufr data into IODA format
yaml_file = f"bufr_snocvr.yaml"
if not os.path.isfile(f"{os.path.join(localconf.DATA, yaml_file)}"):
logger.exception(f"{yaml_file} not found")
raise FileNotFoundError(f"{os.path.join(localconf.DATA, yaml_file)}")
exe = Executable(self.task_config.BUFR2IODAX)
exe.add_default_arg(os.path.join(localconf.DATA, f"{yaml_file}"))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a suggestion to remove some repetition here. Please see the patch on this file if you could kindly test.
diff.txt

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tested this, and it worked perfect. The updates were committed to the repo. Thanks @aerorahul for your suggestions.

logger.info(f"Executing {exe}")
try:
exe()
Expand Down