[Bug]: Dimension Labels and Expandable Datasets Edge Cases

[mavaylon1]

What happened?

1. Within __get_dimension_labels_from_spec (or __get_spec_info depending on if VectorData Refactor Expandable #1158 has been merged) uses get_data_shape to get the shape. But this does not work when we have dtype defined:

data = [(1, 2, 3.)]
dtype=[('x', '<u4'), ('y', '<u4'), ('weight', '<f4')])

get_data_shape will return (1,3)
However, when writing the data we use this for the shape:

elif isinstance(dtype, np.dtype):
            data_shape = (len(data),)

which is (1,)

2. Both VectorData Refactor Expandable #1158 and the current implementation of dimension_labels do not support the case of "untyped, named dataset":

elif spec.data_type_def is None and spec.data_type_inc is None:  # untyped, named dataset
                if spec.name in builder.datasets:
                    sub_builder = builder.datasets[spec.name]
                    self.logger.debug("        Retrieving existing DatasetBuilder '%s' for spec name %s and adding "
                                      "attributes" % (sub_builder.name, repr(spec.name)))
                else:
                    self.logger.debug("        Converting untyped dataset for spec name %s to spec dtype %s"
                                      % (repr(spec.name), repr(spec.dtype)))
                    try:
                        data, dtype = self.convert_dtype(spec, attr_value)
                    except Exception as ex:
                        msg = 'could not convert \'%s\' for %s \'%s\''
                        msg = msg % (spec.name, type(container).__name__, container.name)
                        raise BuildError(builder, msg) from ex
                    self.logger.debug("        Adding untyped dataset for spec name %s and adding attributes"
                                      % repr(spec.name))
                    sub_builder = DatasetBuilder(spec.name, data, parent=builder, source=source, dtype=dtype)
                    builder.set_dataset(sub_builder)

Steps to Reproduce

Above.

Traceback

No response

Operating System

macOS

Python Executable

Conda

Python Version

3.12

Package Versions

No response

[mavaylon1]

@rly We talked about 1), but I did want to point out 2).