-
I want to split my collection, and shard by month. I try to use named vector to do this. That's the steps I do
But it fails and I get the response
So how can I add named vector to exsited collection? I want to split the whole collection to shards because I want to use different params on shards. For example, vector and hnsw indexes of data in recently 3 month should be loaded in RAM, and others should be stored on disk to save memory. Can I do it in another way if shard by named vector is not a solution. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
At this time, it is not possible to add a new named vector to an existing collection. Nor is it possible to shard dynamically, but who knows what t he future might hold. What about this approach: partitioning the data over two collections? Both collections would have just one vector type. One is for recent/hot data, the other is for old/cold data. You configure the hot collection to always live in RAM, and configure the cold collection to always live on disk. You may then set up some form of cron job to transfer old points from hot to cold on a daily basis. You can keep track of the age of all points by adding payload having a timestamp. This is of course a very manual approach, but it at least gives you control over the situation. You'll easily be able to tune both collections and the period of points you transfer. Maybe this documentation page can give you another insight as well: https://qdrant.tech/documentation/tutorials/multiple-partitions/ |
Beta Was this translation helpful? Give feedback.
At this time, it is not possible to add a new named vector to an existing collection. Nor is it possible to shard dynamically, but who knows what t he future might hold.
What about this approach: partitioning the data over two collections? Both collections would have just one vector type. One is for recent/hot data, the other is for old/cold data. You configure the hot collection to always live in RAM, and configure the cold collection to always live on disk. You may then set up some form of cron job to transfer old points from hot to cold on a daily basis. You can keep track of the age of all points by adding payload having a timestamp.
This is of course a very manual approach, but it at…