GRPC error of Resource exhausted while searching

[rsrs2013]

I have inserted the SIFT 1M dataset into milvus. It has 128 dimensional vectors. Then I am searching the top - 5k vectors in it. I have created the IVF_PQ index as below. I am getting Resource exhausted error when searching the vectors. The below is the error.
what is the way around ? Thanks in advance.
I am able to set TOP_K limit until 4k. But for 5k, it fails. I have done batch inserting. Also in the discussions, I could only see GRPC error while inserting and couldn't find a discussion about the error during the search.

grpc RpcError: [search], <_InactiveRpcError: StatusCode.RESOURCE_EXHAUSTED, grpc: trying to send message larger than max (279317012 vs. 268435456)>, <Time:{'RPC start': '2024-12-20 23:27:34.115136', 'gRPC error': '2024-12-20 23:27:54.315166'}>`

I have used the below code to search.

index_params = MilvusClient.prepare_index_params()
    index_params.add_index(
            field_name = "vector",
            index_type = "IVF_PQ",
            metric_type = "L2",
            index_name="vector_index",
            params = {"nlist": 128, "m":16}
            )
    #index_par

   # Define search parameters
   search_params = {
        "metric_type": "L2",  # Use L2 distance for nearest neighbor search
        "params": {"nprobe": 10}  # Adjust nprobe as needed
    }
    
  result = collection.search(
        data=qvecs,          # Query vectors
        anns_field="vector",         # Field to search (your vector field name)
        param=search_params,         # Search parameters
        output_fields=output_fields, 
        limit=5000,                    # Top 
        expr=None                    # Optional expression for filtering (e.g., using other fields)
    )
    

[yhmo]

By default, the limitation for RPC transferring:

• 64MB from client to server
• 256MB from server to client

The limitation is defined in the milvus.yaml:

proxy:
  grpc:
    serverMaxSendSize: 268435456 # The maximum size of each RPC request that the proxy can send, unit: byte
    serverMaxRecvSize: 67108864 # The maximum size of each RPC request that the proxy can receive, unit: byte
    clientMaxSendSize: 268435456 # The maximum size of each RPC request that the clients on proxy can send, unit: byte
    clientMaxRecvSize: 67108864 # The maximum size of each RPC request that the clients on proxy can receive, unit: byte

If the "output_fields" contains vector field name and long-length varchar field name, the returned data size might be larger than 256MB then you will get this error.

You can change the milvus.yaml to enlarge the limitation. Just note, the max size of a single RPC message is 2GB, which is hard code limited by protobuf.

[xiaofan-luan]

less nq, or less topk and less output fields.

64M is already too large and you can not retrieve that large result set in one rpc

[rsrs2013]

Thank you @xiaofan-luan