Understanding Ranged Search

[ivhacks]

Hi everyone, I'm new to Milvus, vector DBs, most AI related stuff. I'm trying to get up to speed using a toy exercise. My goal is to create a 3-dimensional collection that stores colors, and be able to identify all entries similar to a particular color by using a ranged search. I'm stuck on getting Milvus to search by radius. When I try, it just returns everything in my collection, or up to limit entries.

Since I'm going to make video tutorials on this stuff as I learn it, I've got a nifty Blender setup to illustrate this. I'm trying to do a search with radius 120 from point [0, 255, 0] (green).

I'm running Milvus 2.4.5, pulled from Dockerhub a few days ago.
Here's what I've done:

Create collection

curl -s --request POST "http://localhost:19530/v2/vectordb/collections/create" \
--data-raw '{
    "collectionName": "colors",
    "dimension": 3,
    "metricType": "L2",
    "autoId": true
}'

Add items

curl --request POST "http://localhost:19530/v2/vectordb/entities/insert" --data-raw '{
    "data": [
        {"vector": [0.0, 255.0, 0.0]}
    ],
    "collectionName": "colors"
}'

And so on until the collection contains:

[0.0, 255.0, 0.0]
[0.0, 255.0, 55.0]
[100.0, 255.0, 0.0]
[0.0, 200.0, 200.0]
[255.0, 100.0, 0.0]
[255.0, 0.0, 0.0]

These are the 6 points shown in the image.

Load collection

curl -s --request POST "http://localhost:19530/v2/vectordb/collections/load" --data-raw '{
    "collectionName": "colors"
}'

I don't quite get why this is needed, but no problems yet.

Perform search

curl -s --request POST "http://localhost:19530/v2/vectordb/entities/search" --data-raw '{
    "collectionName": "colors",
    "data": [
        [0, 255, 0]
    ],
    "outputFields": ["id", "vector"],
    "searchParams": {
    "metricType": "L2",
    "params": {
            "radius": 14400
        }
    }
}' | jq

I'm expecting to get [0, 255, 0], [100, 255, 0], and [0, 255, 55] as in the picture. But it just returns all 6 points.
14400 is $120^2$, since according to this, https://youtu.be/6yLmbL1o3Ow?t=934 Milvus omits the square root part of the distance formula to increase performance. I've tried with and without specifying the L2 metric type in the search request. I've verified that the collection was properly set up to use the L2 metric using the describe API path.

I've also tried the equivalent search in Python:

from pymilvus import connections, Collection

connections.connect("default", host="192.168.1.17", port=19530)
colors_collection = Collection("colors")
colors_collection.load()

results = colors_collection.search(
    data=[
        [0, 255, 0]
    ],
    anns_field="vector",
    param={
        "radius": 14400
    },
    limit=1000,
    output_fields=["vector"]
)

colors_collection.release()
connections.disconnect("default")

for item in results[0]:
    print(item)

This gives the same result of all 6 entries coming back.

Any help would be appreciated. I'm just a zoomer who copy pastes AI generated code so I don't fully grasp all of this 😇
Thanks in advance!

Relevant docs/discussions for convenience:
https://milvus.io/docs/single-vector-search.md#Range-search
https://milvus.io/api-reference/restful/v2.4.x/v2/Vector%20(v2)/Search.md

#31032
#27160

[yhmo]

Two problems in your script:

1. the param={"radius": xxxxx} should be param={"metric_type": "L2", "params": {"radius": xxxxx}}, see the document of range search: https://milvus.io/docs/single-vector-search.md#Range-search
2. the value 114400 should be 14400

This is the correct script:

from pymilvus import (
    FieldSchema, CollectionSchema, DataType,
    utility, Collection, connections
)
connections.connect("default", host="localhost", port=19530)

vector_field = "color"
schema = CollectionSchema(fields=[
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=False),
    FieldSchema(name=vector_field, dtype=DataType.FLOAT_VECTOR, dim=3),
])

collection_name = "color"
utility.drop_collection(collection_name)
collection = Collection(collection_name, schema)

index_params_ivf={"index_type": "FLAT", "params": {}, "metric_type": "L2"}
collection.create_index(vector_field, index_params=index_params_ivf)
collection.load()

[0.0, 255.0, 0.0]
[0.0, 255.0, 55.0]
[100.0, 255.0, 0.0]
[0.0, 200.0, 200.0]
[255.0, 100.0, 0.0]
[255.0, 0.0, 0.0]
collection.insert({"id": 1, vector_field: [0, 255, 0]})
collection.insert({"id": 2, vector_field: [0, 255, 55]})
collection.insert({"id": 3, vector_field: [100, 255, 0]})
collection.insert({"id": 4, vector_field: [0, 200, 200]})
collection.insert({"id": 5, vector_field: [255, 100, 0]})
collection.insert({"id": 6, vector_field: [255, 0, 0]})

print(collection.query(expr="", output_fields=["count(*)"], consistency_level="Strong"))

results = collection.search(
    data=[
        [0, 255, 0]
    ],
    anns_field=vector_field,
    param={"metric_type": "L2", "params": {"radius": 14400}},
    limit=1000,
    output_fields=[vector_field],
)

print("Search result:")
for item in results[0]:
    print(item)

The expected results:

data: ["{'count(*)': 6}"] 
Search result:
id: 1, distance: 0.0, entity: {'color': [0.0, 255.0, 0.0]}
id: 2, distance: 3025.0, entity: {'color': [0.0, 255.0, 55.0]}
id: 3, distance: 10000.0, entity: {'color': [100.0, 255.0, 0.0]}

[ivhacks]

Thanks for your reply!

The 114400 was a typo. I've fixed it to avoid confusing readers. Sorry about that. I had tried it with the correct number, that wasn't the (main) issue.

The formatting of the search parameters was the issue, now I have it working in Python. Specifying L2 wasn't required because my collection was created to use that metric type by default. It works with or without. Here's my exact fix (mostly for readers):

I tried to replicate this as closely as possible in the curl command, but this still doesn't work (gives all 6):

curl -s --request POST "http://localhost:19530/v2/vectordb/entities/search" --data-raw '{
    "collectionName": "colors",
    "data": [
        [0, 255, 0]
    ],
    "annsField": "vector",
    "searchParams": {
        "params": {
            "radius": 14400
        }
    },
    "limit": 1000,
    "outputFields": ["vector"]
}' | jq

I've also tried variants like changing the order of parameters and specifying L2 explicitly.

Do you recommend any tools or techniques for debugging raw API requests? I've noticed that I can put in nonexistent nonsense fields and it still works and gives a list of results. It seems like there's a simple syntax error here but currently I'm just guess and checking.

Thanks again.

[yhmo]

There is a small bug for restful search api, it cannot accept ""searchParams"".
You can set the rage like this:

curl --location --request POST "http://localhost:19530/v2/vectordb/entities/search" \
--header "Content-Type: application/json" \
--data-raw '{
    "collectionName": "colors",
    "data": [
        [0, 255, 0]
    ],
    "annsField": "vector",
    "params": {
        "radius": 14400
    },
    "limit": 1000,
    "outputFields": [
        "vector"
    ]
}'

[ivhacks]

This works. Thank you so much!!