Recommended way for persistent storage on Qdrant AWS

[Waffleboy]

Hello, would like to ask if theres a recommended way to configure persistent storage on AWS?

Currently we're using qdrant in EKS. One option i can think fo is to configure an EBS volume, attach it to the pod somehow. Tell qdrant to read from that.

Thanks

[timvisee]

EBS is definitely recommended versus other file share approaches.

[timvisee]

I understand by default qdrant puts data in memory though right?

Correct.

So i just need to configure qdrant to load from disk once? or does the from disk option only use disk?

I do not fully understand your question, but let me say the following:

By default Qdrant holds all your data in memory. Though, it also puts everything on disk. That is needed, because if Qdrant would crash, it would be able to read everything from disk again.

You can also set on_disk: true on your vectors, which would put all your vector data on disk permanently. In that case Qdrant may read from disk in all cases when handling vector data.

[Waffleboy]

@timvisee sorry for the word salad, the latter is what I was asking. Thank you!

On that note, I have an EBS loaded at /data. however, by default the qdrant image uses /qdrant. Is there a way to tell qdrant where to store its data on disk (i.e. store on /data)

[timvisee]

Yes, there is. If you're using the Docker image it's easiest just to mount the storage directory to your desired location.

Otherwise you can change the storage path in the configuration, there are two ways of doing so:

• Change storage.storage_path: /data in config.yaml
• Set the QDRANT__STORAGE__STORAGE_PATH=/data environment variable

[Waffleboy]

Mounting the ebs volume to /qdrant results in the files not being updated when changes are made. Docker doesnt override the ones in the volume. https://stackoverflow.com/questions/74450272/why-does-a-file-within-a-docker-volume-not-get-overwritten

[Waffleboy]

I'm still experiencing some difficulty with this.

I've edited config.yml to be

storage:
  storage_path: /data

We're using kube manifests as below. It doesnt seem like the data is being stored? everytime the build is recreated, even though the mount is present at /data, the data is empty.

Persistant Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: qdrant-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 2
  csi:
    driver: ebs.csi.aws.com
    fsType: ext4
    volumeHandle: <vol id redacted>
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.ebs.csi.aws.com/zone
              operator: In
              values:
                - ap-southeast-1a

persistent vol claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  storageClassName: "" # Empty string must be explicitly set otherwise default StorageClass will be set
  volumeName: qdrant-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2