High memory consumption

[arouene]

Hello,

We have a Danswer host installed with docker. The host have 64GB of RAM, but vespa keeps getting killed for OOM. And then while it is restarted, the backend cannot reach again for vespa.

Memory:

# free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        42Gi       2.6Gi       148Mi        18Gi        20Gi
Swap:             0B          0B          0B

Vespa getting killed for OOM:

[Tue Dec 10 17:20:04 2024] oom_reaper: reaped process 163083 (vespa-proton-bi), now anon-rss:0kB, file-rss:208kB, shmem-rss:0kB

Backend cannot reach vespa anymore:

ERROR:    12/11/2024 09:29:58 AM       handle_regular_answer.py  269: [Channel ID: D07EXAZES4U] Unable to process message - did not successfully answer in 5 attempts
Traceback (most recent call last):
  File "/app/danswer/document_index/vespa/chunk_retrieval.py", line 303, in query_vespa
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://index:8081/search/'

If I restart the backend container it's working again.

Is it normal for Danswer to require more than 64GB of memory?

[rkuo-danswer]

Hi Aroune, depends entirely on the number and size of the documents you are indexing. Could you provide us with some more context?