High memory usage despite memory mapping

[abhinavkulkarni]

Hi,

I have a sample dataset of 2M vectors with 512 dimensions, each vector being a float32. I also have an id field that is int64.

So, the size of raw vectors is 2M * 512 * 4 = 4GB.

The collection is memory-mapped.

I have 3 such collections with FLAT, IVF_FLAT, and HNSW indexes on the vector field.

1. Simply loading the collection consumes 4GB of memory, despite being memory-mapped.
2. Querying post-loading consumes up to 5-6GB of memory, despite being memory-mapped.

Releasing the collection does clear the memory - so clearly the data resides on the disk.

Pleaes note queryNode.mmap.mmapEnabled is set to true in values.yaml.

This is how I'm creating the collections:

index_type, params = 'FLAT', {}
# index_type, params = 'IVF_FLAT', {'nlist': 2000}
# index_type, params = 'HNSW', {'M': 32, 'efConstruction': 200}

# Define a collection schema
eid_field = FieldSchema(name="eid", dtype=DataType.INT64, is_primary=True, description="embedding id")
embedding_field = FieldSchema(name="content_embedding", dtype=DataType.FLOAT_VECTOR, dim=512, description="content embedding")

# Set enable_dynamic_field to True if you need to use dynamic fields. 
schema = CollectionSchema(fields=[eid_field, embedding_field], auto_id=False, enable_dynamic_field=True, description=f"{index_type} collection")

# Define index parameters
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="eid",
    index_type="STL_SORT"
)

index_params.add_index(
    field_name="content_embedding", 
    index_type=index_type,
    metric_type="L2",
    params=params
)

collection_name = f"benchmarking_{index_type}"
params_str = '_'.join([f"{k}_{v}" for k,v in sorted(params.items())])
if len(params_str)>0:
    collection_name += f"_{params_str}"
if not client.has_collection(collection_name=collection_name):
    client.create_collection(
        collection_name=collection_name,
        dimension=512,  # The vectors we will use in this demo has 512 dimensions
        schema=schema,  # The collection schema we defined above
        index_params=index_params,
        properties={'mmap.enabled': True}
    )

Here are the peak CPU/memory usage while ingesting 2M vector data:

Simply loading (and subsequently releasing) these collections has the following peak CPU/memory usage:

Here are the peak CPU/memory usage while querying 100K queries (batch_size=100):

This is a test setup, so we just have one replica count for index/data/query nodes. But, in production, we plan to have a distributed cluster with multiple replicas.

For the production use case, we have collections with 100M vectors, so we cannot have a scenario where all the data is loaded in the memory despite memory mapping, as that will be cost-prohibitive.

PS: Please note, I'm only memory mapping the data and not the index.

Thanks!

[abhinavkulkarni]

I have read other related discussions such as #33721, but it is not clear what needs to be done to bring the memory usage down.

[yhmo]

Basically, milvus is an in-memory database. In-memory index can get the best search performance.
If your memory capacity is insufficient for FLAT/IVF_FLAT/HNSW, you can choose IVF_SQ8/IVF_PQ/DISKANN.
So far as I know, mmap doesn't help to reduce memory usage for in-memory index, it is mainly for DISKANN and scalar fields.

[abhinavkulkarni]

Hey @yhmo,

Basically, milvus is an in-memory database

This link says that either or both of data and index files can be memory-mapped.

you can choose IVF_SQ8/IVF_PQ/DISKANN

Out application is extremely sensitive to precision, so cannot go below fp16.

So far as I know, mmap doesn't help to reduce memory usage for in-memory index

So, does that mean that in order to see benefits - both data and index need to be memory mapped?

Thanks!

[xiaofan-luan]

1. 4GB is not a very large memory usage. most likely those on memory overhead. I'd suggest you directly test 10M or more.
2. with mmap, we still try to fully utilize as much as memory to improve performance. You can retry to insert more data until it get OOM to evaluate the maximum capacity
3. search will cost extra memories. With larger topk and batch it could take you memories.

To utilize milvus, we recommend to start from 8GB memory and it's better to use 32GB to evaluate memory usage. Loading data, growing segment, scalar data, search will all cost some memories.

[abhinavkulkarni]

Thanks, I am able to make memory mapping work on larger collections.

I added the following section in values.yaml:

extraConfigFiles:
  user.yaml: |+
    queryNode:
      enableDisk: true
      cache:
        memoryLimit: 536870912
      mmap:
        mmapEnabled: true
        vectorField: true
        vectorIndex: true
        scalarField: true
        scalarIndex: true
        growingMmapEnabled: true

Secondly, while loading the collection, I make sure that both the collection and the index is loaded in memory-mapped fashion:

collection = Collection(name=collection_name)
collection.set_properties({'mmap.enabled': True})
collection.alter_index(
    index_name="embedding",
    extra_params={"mmap.enabled": True}
)
client.load_collection(collection_name=collection_name)

By setting querynode resources.limits.memory to 4Gi, I was able to query a collection of 8M float32 vectors seamlessly with both IVF_FLAT and HNSW.

Thanks!