numpy2.1 + strings caussing errors in assignment

[hmaarrfk]

What happened?

import numpy as np; import xarray as xr
data = np.full((5,), fill_value='nothing')
da = xr.DataArray(data)
values = np.asarray(['foo', 'bar'])
da[1:2] = values[0]

Compatibility matrix with xarray 2024.9.0

• numpy 1.26: works
• numpy 2.0.2: works
• numpy 2.1.1: does not work

I have even tested with and without #9403

import xarray as xr
from packaging.version import Version

if Version(np.__version__) > Version('2.1.0'):
    from xarray.core.variable import as_compatible_data as _as_compatible_data_original
    if not isinstance(_as_compatible_data_original(np.float64(1.)), np.ndarray):
        warn("Monkey patching xarray.core.variable.as_compatible_data "
             "to return numpy arrays.")
        from functools import wraps

        @wraps(_as_compatible_data_original)
        def as_compatible_data(*args, **kwargs):
            from xarray.core.variable import cast
            data = _as_compatible_data_original(*args, **kwargs)
            """Convert obj to a numpy array."""
            if not isinstance(data, np.ndarray | np.generic) and (
                hasattr(data, "__array_function__") or hasattr(data, "__array_namespace__")
            ):
                return cast("T_DuckArray", data)
            return data
        xr.core.variable.as_compatible_data = as_compatible_data

What did you expect to happen?

for it to work

Minimal Complete Verifiable Example

as above

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

# packages in environment at /home/mark/miniforge3/envs/xr:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
libblas                   3.9.0           24_linux64_openblas    conda-forge
libcblas                  3.9.0           24_linux64_openblas    conda-forge
libexpat                  2.6.3                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgfortran               14.1.0               h69a702a_1    conda-forge
libgfortran-ng            14.1.0               h69a702a_1    conda-forge
libgfortran5              14.1.0               hc5f4f2c_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
liblapack                 3.9.0           24_linux64_openblas    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libsqlite                 3.46.1               hadc24fc_0    conda-forge
libstdcxx                 14.1.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.1.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
numpy                     2.1.1           py312h58c1407_0    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.3           py312hf9745cd_1    conda-forge
pip                       24.2               pyh8b19718_1    conda-forge
python                    3.12.4          hb3fe705_101_cpython    ramonaoptics
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
setuptools                74.1.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
tk                        8.6.13          noxft_h25ccc98_102    mark.harfouche
tzdata                    2024a                h8827d51_1    conda-forge
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
xarray                    2024.9.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

Anything else we need to know?

$ mamba list --revisions
2024-09-22 23:25:20  (rev 0)
    +_libgcc_mutex-0.1 (conda-forge/linux-64)
    +_openmp_mutex-4.5 (conda-forge/linux-64)
    +bzip2-1.0.8 (conda-forge/linux-64)
    +ca-certificates-2024.8.30 (conda-forge/linux-64)
    +ld_impl_linux-64-2.40 (conda-forge/linux-64)
    +libblas-3.9.0 (conda-forge/linux-64)
    +libcblas-3.9.0 (conda-forge/linux-64)
    +libexpat-2.6.3 (conda-forge/linux-64)
    +libffi-3.4.2 (conda-forge/linux-64)
    +libgcc-14.1.0 (conda-forge/linux-64)
    +libgcc-ng-14.1.0 (conda-forge/linux-64)
    +libgfortran-14.1.0 (conda-forge/linux-64)
    +libgfortran-ng-14.1.0 (conda-forge/linux-64)
    +libgfortran5-14.1.0 (conda-forge/linux-64)
    +libgomp-14.1.0 (conda-forge/linux-64)
    +liblapack-3.9.0 (conda-forge/linux-64)
    +libnsl-2.0.1 (conda-forge/linux-64)
    +libopenblas-0.3.27 (conda-forge/linux-64)
    +libsqlite-3.46.1 (conda-forge/linux-64)
    +libstdcxx-14.1.0 (conda-forge/linux-64)
    +libstdcxx-ng-14.1.0 (conda-forge/linux-64)
    +libuuid-2.38.1 (conda-forge/linux-64)
    +libxcrypt-4.4.36 (conda-forge/linux-64)
    +libzlib-1.3.1 (conda-forge/linux-64)
    +ncurses-6.5 (conda-forge/linux-64)
    +numpy-1.26.4 (conda-forge/linux-64)
    +openssl-3.3.2 (conda-forge/linux-64)
    +packaging-24.1 (conda-forge/noarch)
    +pandas-2.2.3 (conda-forge/linux-64)
    +pip-24.2 (conda-forge/noarch)
    +python-3.12.4 (ramonaoptics/linux-64)
    +python-dateutil-2.9.0 (conda-forge/noarch)
    +python-tzdata-2024.1 (conda-forge/noarch)
    +python_abi-3.12 (conda-forge/linux-64)
    +pytz-2024.1 (conda-forge/noarch)
    +setuptools-74.1.2 (conda-forge/noarch)
    +six-1.16.0 (conda-forge/noarch)
    +tk-8.6.13 (mark.harfouche/linux-64)
    +tzdata-2024a (conda-forge/noarch)
    +wheel-0.44.0 (conda-forge/noarch)
    +xarray-2024.9.0 (conda-forge/noarch)
    +xz-5.2.6 (conda-forge/linux-64)

2024-09-22 23:28:58  (rev 1)
     numpy  {1.26.4 (conda-forge/linux-64) -> 2.0.2 (conda-forge/linux-64)}

2024-09-22 23:29:31  (rev 2)
     numpy  {2.0.2 (conda-forge/linux-64) -> 2.1.1 (conda-forge/linux-64)}

Environment

conda-forge as above

[hmaarrfk]

Opps. here is the traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mark/miniforge3/envs/xr/lib/python3.12/site-packages/xarray/core/dataarray.py", line 917, in __setitem__
    self.variable[key] = value
    ~~~~~~~~~~~~~^^^^^
  File "/home/mark/miniforge3/envs/xr/lib/python3.12/site-packages/xarray/core/variable.py", line 878, in __setitem__
    value = value.set_dims(dims).data
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/miniforge3/envs/xr/lib/python3.12/site-packages/xarray/util/deprecation_helpers.py", line 143, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/miniforge3/envs/xr/lib/python3.12/site-packages/xarray/core/variable.py", line 1387, in set_dims
    expanded_data = self.data[indexer]
                    ~~~~~~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'tuple'

[max-sixty]

One quick point — seems to be an issue with numpy.str_ — da[1:2] = 'foo' works OK...

[max-sixty]

#9536

[keewis]

that's funny. I can't reproduce this with the current state of #9403 (even on numpy=2.1.1), which makes sense to me because np.str_ is a subclass of np.generic and thus will be caught by the condition I added there.

[hmaarrfk]

@keewis you are correct #9403 (git describe v0.17.0-2074-g9aed88d4) does resolve things.

not sure why my monky patch doesn't catch it, but I guess that why we still restrict numpy<2 in the wild.

Thanks for looking into it!