Add a Multi-Vector Similarity Function

[vigyasharma]

This is a small first change towards adding support for multi-vectors. We start with adding a MultiVectorSimilarityFunction that can handle (late) interaction across multiple vector values.

This is the first of a series of splits for the larger prototype PR #13525

[vigyasharma]

I am thinking we can leverage the NONE aggregation (in #13525) for non-ColBERT passage vector use-cases, by making each graph node correspond to a single value in the multi-vector i.e. index time aggregation becomes "none". The resultant graph would be similar to what we construct with parent-child docs today, while flat storage with multi-vectors could allow for aggregated similarity checks at query time. This could help with recall while making mutli-vector usage easier to use (no overquery or index time joins).

It'll need some work: a mechanism to address each vector value directly, and corresponding changes in VectorValues. I'm thinking maybe an "ordinal" for the multi-vector, and a "sub-ordinal" for values within the multi-vector. Both ints can be packed into a long for node value?

Since I haven't chalked out all the details yet, I decided to remove the NONE aggregation for now, and focus first on the ColBERT use case. Will go with the isMultiVector flag in FieldInfos to identify multi-vector storage requirements (and keep NONE aggregation free to this lever).

cc: @cpoerschke , @benwtrent

[dungba88]

If we add another compare method with start and end indexes for both inner and outer, I guess we won't need to copy the array?

[vigyasharma]

Right, but it needs to go all the way down to VectorUtilSupport, which I think should be a PR of its own.

[dungba88]

It's so unfortunate that Java doesn't support generic primitive array and has to have duplicate code.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!