Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix read netcdf not working on single time step zonal mean data #285

Open
23 tasks
CPKalb opened this issue Feb 27, 2024 · 0 comments
Open
23 tasks

Bugfix read netcdf not working on single time step zonal mean data #285

CPKalb opened this issue Feb 27, 2024 · 0 comments
Assignees
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle alert: NEED MORE DEFINITION Not yet actionable, additional definition required type: bug Fix something that is not working

Comments

@CPKalb
Copy link

CPKalb commented Feb 27, 2024

Describe the Problem

METdataio read_netcd work for zonal mean data when an array of times is input. However, with a single time step, the script fails with the following error:
Traceback (most recent call last):
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 285, in decode_cf_datetime
dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 218, in _decode_datetime_with_pandas
raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'julian', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 184, in _decode_cf_datetime_dtype
result = decode_cf_datetime(example_value, units, calendar, use_cftime)
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 287, in decode_cf_datetime
dates = _decode_datetime_with_cftime(
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 208, in _decode_datetime_with_cftime
cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
File "src/cftime/_cftime.pyx", line 617, in cftime._cftime.num2date
File "src/cftime/_cftime.pyx", line 414, in cftime._cftime.cast_to_int
OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/conventions.py", line 551, in decode_cf_variables
new_vars[k] = decode_cf_variable(
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/conventions.py", line 397, in decode_cf_variable
var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 716, in decode
dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/coding/times.py", line 194, in _decode_cf_datetime_dtype
raise ValueError(msg)
ValueError: unable to decode time units 'days since 2018-02-12' with "calendar 'julian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/glade/u/home/kalb/stratosphere/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py", line 112, in
main()
File "/glade/u/home/kalb/stratosphere/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py", line 47, in main
dsO = file_reader.read_into_xarray(obs_infiles)[0]
File "/glade/u/home/kalb/stratosphere/METdataio/METreadnc/util/read_netcdf.py", line 70, in read_into_xarray
file_data = xr.open_dataset(file)
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/backends/api.py", line 541, in open_dataset
backend_ds = backend.open_dataset(
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 592, in open_dataset
ds = store_entrypoint.open_dataset(
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/backends/store.py", line 35, in open_dataset
vars, attrs, coord_names = conventions.decode_cf_variables(
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/xarray/conventions.py", line 562, in decode_cf_variables
raise type(e)(f"Failed to decode variable {k!r}: {e}")
ValueError: Failed to decode variable 'time': unable to decode time units 'days since 2018-02-12' with "calendar 'julian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

Expected Behavior

It should successfully read data with a single time step or an array of times

Environment

Describe your runtime environment:
*1. derecho
*2. linux
3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Organization level Project for support of the current coordinated release
  • Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of main_<Version>.
    Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
  • Fix the bug and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into main_<Version>.
    Pull request: bugfix <Issue Number> main_<Version> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues
    Select: Organization level software support Project for the current coordinated release
    Select: Milestone as the next bugfix version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch.
    Branch name: bugfix_<Issue Number>_develop_<Description>
    Pull request: bugfix <Issue Number> develop <Description>
    Select: Reviewer(s) and Development issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Close this issue.
@CPKalb CPKalb added alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED MORE DEFINITION Not yet actionable, additional definition required type: bug Fix something that is not working alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Feb 27, 2024
@hankenstein2 hankenstein2 added this to the METdataio-3.0.0 milestone Feb 28, 2024
@hankenstein2 hankenstein2 self-assigned this Apr 3, 2024
@bikegeek bikegeek moved this from 🟢 Ready to 📖 Backlog in METplus-Analysis-6.1.0 Development Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle alert: NEED MORE DEFINITION Not yet actionable, additional definition required type: bug Fix something that is not working
Projects
Status: 🩺 Needs Triage
Development

No branches or pull requests

3 participants