what happens in the case of failure while uploading the data

[vihariazure]

hi team

below are the list of questions

i have 1000 records
500 records inserted after that some failure happened

in that case if i re-run the code again, only that left over 500 records are inserted or 1000 records will be uploading to collection

2. what about index creation

for example

lets say i have built the index by 1k vectors. i passed query and got the results. now i have inserted another 100 records.
during 100 records insertion, now the whole index will be rebuilt again using 1100 vectors or only this 100vectors are inserted to existing index.(no reindex will happen)

while inserting , can we pass the query

while the query is passed only 1000 record will considered or 1000 + (100 not inserted into index ) is also considered

[timvisee]

1. in that case if i re-run the code again, only that left over 500 records are inserted or 1000 records will be uploading to collection

All the points you send to the API will be upserted. So it depends on how you upsert those points.

If your upsert logic starts from the very first point again if it failed at some point, all points will be (re)upserted. If your upsert logic continues from where it failed, it'll only upsert those.

Note that if you upsert a point with the same ID, it'll overwrite the point that is already there.

That means that if you upsert all points again using the same IDs, it'll likely replace 500 existing ones and upsert 500 new ones.

Note that our recommendation is to send all operations that fail again, until they do succeed.

2. while inserting , can we pass the query

Yes. Points you upsert are immediately available.

Some of them may not be indexed straight away, but even if they aren't, you'll still be searching through them.

[vihariazure]

thanks for the reply

in milvus there is concepts called segments , when each segment reaches the limit , each segment create the index(independently). similarly we have any concept in the qdrant. or we have only one index for whole inserted data. if any resources available please share

[timvisee]

Qdrant also uses segments internally, and shards if you have configured it to use multiple shards.

This page roughly describes it: https://qdrant.tech/documentation/concepts/indexing/

I wouldn't be worried to much about how indexing is implemented, unless you have a specific question you'd like to get an answer to.

[vihariazure]

thanks for the rply

iam using data bricks to insert data to qdrant. in the vm qdrant is running

i gone through below link which describes about shards

https://qdrant.tech/documentation/guides/distributed_deployment/#sharding

below is my data bricks cluster configuration

for 1m regards of vector it took 20m (with batch size 2000)

i want to speed up insertion process by parallel ingesting the data

how shades do i need to set or how to use worker nodes also

[vihariazure]

@timvisee @dzhao @bazhenov @yarikoptic

any update on it