YYYY-12-31 issue when selecting a period from a daily/monthly files (360Day calendar)

[bonfils2]

Some models fail when we specify December 31st as the ending index of a time slice.
This is an issue specific to the files using a 360Days calendar.

Example:
fn=’/p/css03/esgf_publish/CMIP6/ScenarioMIP/MOHC/UKESM1-0-LL/ssp585/r3i1p1f2/day/pr/gn/v20190813/’
ds=xc.open_mfdataset(fn+'*.nc',add_bounds=True)
ds=ds.sel(time=slice('2077-01-01','2078-12-31'))
--> ValueError: invalid day number provided in cftime.Datetime360Day(2078, 12, 31, 0, 0, 0, 0, has_year_zero=True)

What we found:
For monthly files, a solution is to specify: YYYY-12: ds=ds.sel(time=slice('2077-01','2078-12'))

For daily files, there are two workaround solutions:
Solution 1: opening the next January first and drop it.
ds=ds.sel(time=slice('2077-01-01','2079-01-01'))
ds=ds.isel(time=slice(0,ds.pr.shape[0]-1))

Solution 2: select the period this way:
ds = ds.where(((ds['time.year'] >= 2076) & (ds['time.year'] < 2078)), drop=True)

Any other ideas?

[pochedls]

Note that cdms2 had syntax to denote whether something should be selected "inclusive/closed" or "exclusive/open". I think something like this could be added with a function / class (e.g., ds.subset_time(start, end, openclosed='co') or da.subset_time(start, end, openclosed='co')).

[bonfils2]

That would be very useful.
In cdms2, this code was something like this: s1=f(‘tas’,time=('2076-1-01','2079-1-01','con')).

[jypeter]

This very useful cdms2 feature is summarized on page 9 (The mysterious “third argument”) of the Unlocking_CDATs_Best_Kept_Secrets.pdf document that you can access from #170, and in other documents as well (probably cdms5.pdf)

Note that I have used this argument a lot for selecting a longitude range with calls like tas = f('tas', longitude=(-180, 180, 'co')), especially when dealing with non CMORized models from different institutes. This ensured that I ended up with longitudes in the [-180, 180[ without having the -180 longitude duplicated (at +180), even if the original model data was stored on [0, 360[

Can xarray do that (i.e reorder longitudes on the fly when loading data, knowing that longitudes have a period of 360)?

[pochedls]

There is a lon_orient argument that can be used with open_dataset() and a separate swap_lon_axis() function. Note there is one bug and a feature request with this functionality.

So far all of the functionality around selection/indexing has been left to xarray, but this thread makes it clear that it may be useful to add xcdat routines.

[chengzhuzhang]

I think @tomvothecoder discovered this same issue when refactor e3sm diags code, that specifying year range would result one month short with xcdat/xarray applied to e3sm data.

[acordonez]

I gave the inclusive parameter a try:

xr.date_range(start=cftime.datetime(1970,1,5),
              end = cftime.datetime(1971,1,1),
              freq = "D",
              inclusive="left",
              calendar="360_day",
              use_cftime=True)

The result does drop 1971-1-1, but it includes 1970-12-31!

CFTimeIndex([1970-01-05 00:00:00, 1970-01-06 00:00:00, 1970-01-07 00:00:00,
             1970-01-08 00:00:00, 1970-01-09 00:00:00, 1970-01-10 00:00:00,
             1970-01-11 00:00:00, 1970-01-12 00:00:00, 1970-01-13 00:00:00,
             1970-01-14 00:00:00,
             ...
             1970-12-22 00:00:00, 1970-12-23 00:00:00, 1970-12-24 00:00:00,
             1970-12-25 00:00:00, 1970-12-26 00:00:00, 1970-12-27 00:00:00,
             1970-12-28 00:00:00, 1970-12-29 00:00:00, 1970-12-30 00:00:00,
             1970-12-31 00:00:00],
            dtype='object', length=361, calendar='standard', freq='D')

I've looked into the output index and it's making a standard calendar, eg Feb has 28 days, March has 31.
If someone sees a way I should fix my command, let me know and I'll give it a try.

[pochedls]

Does this inclusive parameter only apply when generating your own time axis (via xr.date_range)?

I can't tell if this parameter could be applied to selecting data (the original question) or if this related functionality is just being flagged as related/relevant (based on some experience using this functionality with E3SM).

[tomvothecoder]

I mentioned xr.date_range with the inclusive parameter because it seems similar to cdms2's subsetting openclosed parameter.

I haven't investigated whether the xr.date_range inclusive parameter exists for subsetting, which would replicate cdms2. Maybe somebody else can investigated this if I don't get to it anytime soon.

[pochedls]

I tried (and failed) to find a parameter/option like this

[gkb999]

@pochedls @tomvothecoder
Hi, I have a similar issue (if not exact)
I don't want to be opening another issue on this.
I used xcdat to generate daily climatology,

but, the time axis seems to be disturbed. Although it shows 365 days value (below -screenshot 1), while plotting, I get the error as
\envs\xcdat_env\Lib\site-packages\nc_time_axis\__init__.py:359: CFWarning: this date/calendar/year zero convention is not supported by CF lower = cftime.num2date(vmin, self.date_unit, calendar=self.calendar) \xcdat_env\Lib\site-packages\nc_time_axis\__init__.py:393: CFWarning: this date/calendar/year zero convention is not supported by CF dt = cftime.datetime(

and the clim graph looks like this: see the time axis

Not just the daily one, even monthly one seems disturbed

Please suggest how can the axis be corrected

[acordonez]

Just adding my example of working with this issue in the 360 day calendar. In this case, I'm calculating the max rolling 5-day precipitation mean for each year, and I only want days within that calendar year to be used for its average. I.E., when calculating the max for 1970, I used the 5-day rolling mean starting on Jan 5 1970 and ending on the final day of 1970.

I've also run into this problem when getting the DJF max during leap years in the "noleap" calendar.

import xcdat as xc
import xarray as xr
import numpy as np
import cftime
import datetime

##############
# Intro
##############
# We want the max 5 day rolling mean value of precipitation
# for each year. In this case, I want this value to be
# calculated using only days from that year. I.E., the 
# first rolling mean date counted in 1970 is 1/5/1970
# and the final is 12/30/1970 in the 360_day calendar

##############
# Create data
##############
times = xr.cftime_range(start="1970-01-01", end="1999-12-30", freq="D", calendar="360_day", name="time")
lats = np.array([0.5,1.5,2.5,3.5,4.5])
lons = np.array([5.,6.5,8.,9.5])
np.random.seed(456)
values = np.random.randn(len(times),5,4)
ds = xr.Dataset({'pr': xr.DataArray(
                    data   = values,   # enter data here
                    dims   = ['time',"lat","lon"],
                    coords = {'time': times, "lat": lats, "lon": lons},
                    attrs  = {
                        '_FillValue': -999.9,
                        'units'     : ''
                        }
                    )})

ds['time'] = ds.time.assign_attrs(encoding="360_day")
ds = ds.bounds.add_missing_bounds()
ds['pr'].time.encoding = {'calendar': "360_day"}   
ds.to_netcdf("test_ds.nc")

##############
# Set up
##############
ds = xc.open_dataset("test_ds.nc")
year_beg = ds.isel({"time": 0}).time.dt.year.item()
year_end = ds.isel({"time": -1}).time.dt.year.item()
year_range = np.arange(year_beg,year_end+1,1)

cal = ds.time.encoding["calendar"]

fx_rolling =ds.pr.rolling(time=5).mean()

del_one_d = datetime.timedelta(days=1)
del_zero_d = datetime.timedelta(days=0)

##############
# Example 1
##############
# Here's the working way to get the max value for these dates
# I make a list of the dates to select, then pass
# those to the xr.sel() function.

# Why subtract 0 days from the start time? Easy way to 
# force it into the same datetime data type as the end time - 1 day.

date_range = [xr.cftime_range(
                              start=cftime.datetime(year,1,5,calendar=cal)-del_zero_d,
                              end = cftime.datetime(year+1,1,1,calendar=cal)-del_one_d,
                              freq='D',
                              calendar=cal) for year in year_range]
date_range = [item for sublist in date_range for item in sublist]
fx_ann_max = fx_rolling.sel(time=date_range).groupby("time.year").max(dim="time")

##############
# Example 2
##############
# As a counter example, this will not work with this calendar
date_range = [xr.cftime_range(
                              start=cftime.datetime(year,1,5,calendar=cal),
                              end = cftime.datetime(year,12,31,calendar=cal),
                              freq='D',
                              calendar=cal) for year in year_range]
# This throws the following:
# ValueError: invalid day number provided in cftime.datetime(1970, 12, 31, 0, 0, 0, 0, calendar='360_day', has_year_zero=True)