Grouper, Resampler as public api #8840

dcherian · 2024-03-15T05:16:05Z

Expose UniqueGrouper, BinGrouper and TimeResampler as public API under xarray.groupers
Allow providing these objects to .resample and .groupby
This is a step toward grouping by multiple variables, grouping by a dask array, and unblocking "chunking to a frequency" (Support specifying chunk sizes using labels (e.g. frequency string) #7559)

We do not yet officially support custom Grouper or Resampler objects. I'd like to clear up the API a bit before exposing that.

Consider avoiding IndexVariable (groupby: remove some internal use of IndexVariable #9123)
Closes Update GroupBy constructor for grouping by multiple variables, dask arrays #6610
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

xarray/core/dataarray.py

xarray/tests/test_groupby.py

xarray/core/groupers.py

xarray/core/types.py

dcherian · 2024-06-21T22:36:29Z

This is finally ready for review! Please take a look at the API and docs.

xarray/tests/test_groupby.py

* main: promote floating-point numeric datetimes to 64-bit before decoding (pydata#9182) also pin `numpy` in the all-but-dask CI (pydata#9184) temporarily remove `pydap` from CI (pydata#9183) temporarily pin `numpy<2` (pydata#9181) Change np.core.defchararray to np.char (pydata#9165) (pydata#9166) Fix example code formatting for CachingFileManager (pydata#9178) Slightly improve DataTree repr (pydata#9064) switch to unit `"D"` (pydata#9170) Docs: Add page with figure for navigating help resources (pydata#9147) Add test for pydata#9155 (pydata#9161) Remove mypy exclusions for a couple more libraries (pydata#9160) Include numbagg in type checks (pydata#9159) Improve zarr chunks docs (pydata#9140)

xarray/core/groupers.py

xarray/core/dataset.py

xarray/core/groupers.py

xarray/core/groupby.py

dcherian · 2024-07-11T10:04:31Z

Thanks @keewis and @Illviljan

#9233 completes the deprecation cycles for base and loffset. I'll merge after this one to avoid dealing with a bunch of conflicts.

The mypy failures appear unrelated now.

Illviljan · 2024-07-11T11:31:35Z

These looks related, same problem as before I guess:

xarray/core/dataset.py: note: In member "groupby" of class "Dataset":
xarray/core/dataset.py:10321: error: Incompatible types in assignment (expression has type "Mapping[Hashable, Any]", variable has type "dict[str, Grouper]")  [assignment]
xarray/core/dataarray.py: note: In member "groupby" of class "DataArray":
xarray/core/dataarray.py:6778: error: Incompatible types in assignment (expression has type "Mapping[Hashable, Any]", variable has type "dict[str, Grouper]")  [assignment]

* main: exclude the bots from the release notes (pydata#9235) switch the documentation to run with `numpy>=2` (pydata#9177) `numpy` 2 compatibility in the iris code paths (pydata#9156) `numpy` 2 compatibility in the `netcdf4` and `h5netcdf` backends (pydata#9136) Fix time indexing regression in `convert_calendar` (pydata#9192) Use duckarray assertions in test_coding_times (pydata#9226) Use reshape and ravel from duck_array_ops in coding/times.py (pydata#9225) Cleanup test_coding_times.py (pydata#9223) Only use necessary dims when creating temporary dataarray (pydata#9206) Fix two bugs in DataTree.update() (pydata#9214) Use numpy 2.0-compat `np.complex64` dtype in test (pydata#9217)

dcherian · 2024-07-11T16:57:38Z

I ended up using type: ignore after struggling for a while...

* main: Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" Test main push Revert "Update _typing.py" Update _typing.py Add a `.drop_attrs` method (pydata#8258)

dcherian · 2024-07-17T04:24:22Z

I'll merge tomorrow if there are no comments.

One thing to think about is whether we want UniqueGrouper or CategoricalGrouper or CategoryGrouper.

headtr1ck

Looks good, only couple of minor details.
Feel free to ignore these as well :P

xarray/core/dataarray.py

xarray/core/dataset.py

xarray/core/groupers.py

xarray/core/types.py

xarray/core/groupers.py

Co-authored-by: Michael Niklas <[email protected]>

* main: add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234)

* main: (54 commits) Adding `open_datatree` backend-specific keyword arguments (#9199) [pre-commit.ci] pre-commit autoupdate (#9202) Restore ability to specify _FillValue as Python native integers (#9258) add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234) Allow mypy to run in vscode (#9239) Revert "Test main push" ...

…monotonic-variable * main: (995 commits) Adding `open_datatree` backend-specific keyword arguments (pydata#9199) [pre-commit.ci] pre-commit autoupdate (pydata#9202) Restore ability to specify _FillValue as Python native integers (pydata#9258) add backend intro and how-to diagram (pydata#9175) Fix copybutton for multi line examples in double digit ipython cells (pydata#9264) Update signature for _arrayfunction.__array__ (pydata#9237) Add encode_cf_datetime benchmark (pydata#9262) groupby, resample: Deprecate some positional args (pydata#9236) Delete ``base`` and ``loffset`` parameters to resample (pydata#9233) Update dropna docstring (pydata#9257) Grouper, Resampler as public api (pydata#8840) Fix mypy on main (pydata#9252) fix fallback isdtype method (pydata#9250) Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" ...

dcherian mentioned this pull request Mar 15, 2024

Proof of concept - public Grouper objects #8509

Closed

Illviljan reviewed Mar 21, 2024

View reviewed changes

xarray/core/dataarray.py Outdated Show resolved Hide resolved

xarray/tests/test_groupby.py Outdated Show resolved Hide resolved

dcherian force-pushed the grouper-public-api branch from 3c6d2d5 to 8fc7f5f Compare April 17, 2024 03:44

dcherian commented Apr 17, 2024

View reviewed changes

xarray/core/groupers.py Outdated Show resolved Hide resolved

dcherian commented Apr 17, 2024

View reviewed changes

xarray/core/types.py Outdated Show resolved Hide resolved

dcherian mentioned this pull request Jun 13, 2024

Grouper refactor #9122

Merged

dcherian force-pushed the grouper-public-api branch from 04b433a to 235b553 Compare June 13, 2024 22:55

dcherian mentioned this pull request Jun 13, 2024

groupby: remove some internal use of IndexVariable #9123

Merged

dcherian force-pushed the grouper-public-api branch from b8085f8 to 384818f Compare June 14, 2024 16:50

TomNicholas added the topic-groupby label Jun 14, 2024

dcherian force-pushed the grouper-public-api branch 3 times, most recently from 436997b to 7100907 Compare June 21, 2024 22:08

dcherian added 6 commits June 21, 2024 16:35

Grouper, Resampler as public API

65d500c

Add test

9f5b9fd

Add docs

f338e45

Fix test

09fa705

fix types.

110517a

bugfix

01fbf50

dcherian force-pushed the grouper-public-api branch from 7100907 to 01fbf50 Compare June 21, 2024 22:35

dcherian added the needs review label Jun 21, 2024

dcherian marked this pull request as ready for review June 21, 2024 22:35

dcherian commented Jun 21, 2024

View reviewed changes

xarray/tests/test_groupby.py Outdated Show resolved Hide resolved

dcherian added 3 commits June 28, 2024 10:47

Better binning API

e250895

docs.fixes

0dc1663

dcherian closed this Jun 30, 2024

dcherian reopened this Jun 30, 2024

Illviljan reviewed Jun 30, 2024

View reviewed changes

dcherian added 2 commits July 11, 2024 22:25

Try to fix typing

1142663

dcherian force-pushed the grouper-public-api branch from 12c4316 to c94d9c2 Compare July 11, 2024 16:57

dcherian mentioned this pull request Jul 11, 2024

groupby, resample: Deprecate some positional args #9236

Merged

dcherian force-pushed the grouper-public-api branch from a38206d to c94d9c2 Compare July 11, 2024 17:09

Nicer error

7838d57

dcherian added plan to merge Final call for comments and removed needs review labels Jul 17, 2024

dcherian added 2 commits July 17, 2024 09:34

Try fixing types

cb4faad

dcherian and others added 2 commits July 17, 2024 09:56

fix

792098c

Merge branch 'main' into grouper-public-api

8e1ed58

headtr1ck approved these changes Jul 17, 2024

View reviewed changes

dcherian and others added 6 commits July 17, 2024 20:27

Apply suggestions from code review

8b40396

Co-authored-by: Michael Niklas <[email protected]>

Review comments

a2b0f05

Add whats-new note

d521849

Fix

bf23f3d

Add more types

25c897d

Fix link

1cafa67

dcherian enabled auto-merge (squash) July 18, 2024 03:18

FIx docs

78252b2

dcherian merged commit b55c783 into pydata:main Jul 18, 2024
28 checks passed

dcherian deleted the grouper-public-api branch November 17, 2024 16:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grouper, Resampler as public api #8840

Grouper, Resampler as public api #8840

dcherian commented Mar 15, 2024 •

edited

Loading

dcherian commented Jun 21, 2024

dcherian commented Jul 11, 2024 •

edited

Loading

Illviljan commented Jul 11, 2024

dcherian commented Jul 11, 2024

dcherian commented Jul 17, 2024

headtr1ck left a comment

Grouper, Resampler as public api #8840

Grouper, Resampler as public api #8840

Conversation

dcherian commented Mar 15, 2024 • edited Loading

dcherian commented Jun 21, 2024

dcherian commented Jul 11, 2024 • edited Loading

Illviljan commented Jul 11, 2024

dcherian commented Jul 11, 2024

dcherian commented Jul 17, 2024

headtr1ck left a comment

Choose a reason for hiding this comment

dcherian commented Mar 15, 2024 •

edited

Loading

dcherian commented Jul 11, 2024 •

edited

Loading