Change output of M2 groupby aggregation from a single double column into a structs column
#9899
Conversation
ttnghia
added
3 - Ready for Review
We shouldn't be returning aggregation results that aren't exactly the aggregation requested. If these aggregations are needed, then they should be requested when performing the The caching mechanisms will ensure that no aggregation is redundantly computed. |
| @@ -117,6 +117,14 @@ struct empty_column_constructor { | |||
| 0, make_empty_column(type_to_id<offset_type>()), empty_like(values), 0, {}); | |||
| } | |||
|
|
|||
| if constexpr (k == aggregation::Kind::M2 || k == aggregation::Kind::MERGE_M2) { | |||
| std::vector<std::unique_ptr<column>> child_columns; | |||
There was a problem hiding this comment.
👍
I'm wondering whether this can be phrased differently, as you have suggested before:
auto begin = cudf::make_counting_transform_iterator(0, [](auto i){ return make_empty_column(FLOAT64); });
return make_structs_column(0, std::vector<std::unique_ptr<column>>(begin, begin+4), 0, {});
Thanks. I'm going to work on the Spark plugin side, trying to optimize the computation there. Will get back and close this if the optimization can be done entirely there without cudf work. |
|
This PR has been labeled |
|
This PR has been labeled |
|
@ttnghia were you able to resolve this on the Spark side, or is this still something that we need to add to libcudf? |
This needs to be checked with spark-rapids plugin first but I still don't have bandwidth to work on it yet. Will try again some later time. Thanks. |
Currently,
M2groupby aggregation only outputs a single double type column. During computing them2values, the intermediate results including groupbycountandmeanare all discarded. However, for merging thesem2values, we must have the valuescountandmeanavailable. As such, in Spark plugin, we have to re-compute these values again, which seems to be too inefficient.This PR addresses that issue: The intermediate values
countandmeanare all output together withm2values. In particular, the output result of theM2groupby aggregation now is a structs column containing tuples of (count,mean,m2).