[Feature] Support filter pushdown for Log Tables

[wuchong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, we have supported projection pushdown for Log Tables. Meanwhile, we want to support filter pushdown as well, which can improve query performance and resource efficiency a lot.

This feature is very similar to Kafka Header Filter, but header is not directly accessible in SQL. In SQL language, this is filter pushdown, that automatically optimizes users' where col1 = 'something' condition into a pushdown operation to avoid transferring the unnecessary data.

Solution

We may need to add some metadata/statistics (e.g., min/max) for each Arrow log batch. This allows TabletServers can apply the pushdowned predicate on the metadata without reading and deserializing the Arrow log.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!