Create DataArray with empty .data?

[ghbrown]

I'd like to be able to create a DataArray with dims, coordinates, etc. but no .data field.
If this is possible, what is the easiest way to achieve this?

In case my problem can be solved in an easier way what I'd like is an xarray based solution to track metadata (dims and coordinates specifically) about an object which is naturally described as an xarray.DataArray but cannot be placed in the .data attribute. Hence, I don't want to pay the cost of storing the zeros/NaNs in .data.

[dcherian]

For minimal memory usage: np.broadcast_to(np.array([0]), shape)

Also see https://xarray-schema.readthedocs.io/en/latest/quickstart.html though it doesn't support coordinate values IIUC

[dcherian]

Coordinates is a great idea. I would try that!

[ghbrown]

From my basic tests it seems like the broadcast_to() approach uses that same amount of memory as leaving the data field uninitialized when calling DataArray(). Is this what you expected?
I was hoping for something not proportional to the total number of elements.

[dcherian]

No. nbytes is just wrong. np.broadcast_to creates a view. Only 8 bytes are used:

np.broadcast_to([1], (5,10)).base.nbytes

[ghbrown]

np.broadcast_to creates a view

That makes sense, thanks.

However, when I try to pickle a DataArray with data set via broadcasting it still requires the full amount of memory (as read by du). Is this expected, or have I messed something up?

[TomNicholas]

You're purely in numpy land with that question - take it up with them :)

[TomNicholas]

If this is possible, what is the easiest way to achieve this?

I don't think this is possible. You have to wrap some kind of array (though it can take up very little space as in Deepak's suggestion).

DataArray with dims, coordinates, etc. but no .data field.

The dims of a DataArray are the dims of the Variable it contains, so no variable means no dims. And coordinates on a DataArray cannot have dims that the Variable does not have.

You can however create an xr.Dataset with coordinate variables but no data variables, and the dataset will have the set of all dims present in all coordinates. (But the new xr.Coordinates object stands alone...)

an object which is naturally described as an xarray.DataArray but cannot be placed in the .data attribute

As well as xarray-schema you might also be interested in how virtualizarr works - by creating a dummy array-like object to wrap, and deliberately avoiding creating any in-memory Indexes. This dummy object could store literally nothing if you want it to, it only needs to present a dtype and a shape which obeys broadcasting rules.

[ghbrown]

That worked as expected, thanks!

FWIW it looks like the new scipy.sparse.coo_array is similar to sparse.COO (whose original motivation was to go beyond sparse matrices before scipy supported that). Not sure if this simplifies or complicates life for you all...

[TomNicholas]

FWIW it looks like the new scipy.sparse.coo_array is similar to sparse.COO (whose original motivation was to go beyond sparse matrices before scipy supported that). Not sure if this simplifies or complicates life for you all...

That's cool! But in terms of compatibility with xarray all we care about is whether they follow the requirements listed here, which I'm guessing scipy.sparse.coo_array does not (I'm sure there will be an issue saying this already somewhere - @keewis would know.)

[TomNicholas]

(I'll mark this as answered for now)

[ghbrown]

#7280 yep.

I'm writing a comprehensive answer now that I'll accept if that's okay.

[keewis]

for the sake of completeness (I know I'm kinda late), scipy.sparse has been transitioning from np.matrix-like classes (scipy.sparse.coo_matrix) to np.ndarray-like ones (scipy.sparse.coo_array).

That process has been going on for a while, but I don't think the Array API or __array_ufunc__ and __array_function__ have been implemented, yet (now that it's not based on np.matrix anymore, this has become feasible). See the tracking issue (scipy/scipy#18915), and scipy/scipy#20190 for an implementation attempt.

[ghbrown]

Thanks to @TomNicholas and @dcherian for the many answers and suggestions, this comprehensive answer just puts all the information in one spot for future readers.

Answer

It's not directly possible to have and empty .data attribute, but using a sparse.COO array of the correct shape with no nonzeros results in negligible memory usage. At the time of writing scipy.sparse.coo_array was not supported as an input to DataArray() and gave an error (see #7280).
It may also be possible to use a view of an array by setting
data=np.broadcast_to(np.array([0]), <your shape>)
but when trying to save the resulting xarray.DataArray as a pickle this resulted in memory usage proportional to the size of the array, which I was trying to avoid.

In my case a more appropriate solution was using and storing the DataArray.coords of my data array, since it retains all of the stuff I needed (ordered dims, coordinate information) without having to spoof an array.

[ghbrown]

Random note: pickling DataArray.coords uses the same amount of storage as pickling the original DataArray, which was surprising.

So I end up pickling xarray.Coordinates(coords=DataArray.coords), which is the expected size.