Can't start after 1.222.0 update #14499
Replies: 12 comments 24 replies
-
|
Your database is damaged/broken. Please post the logs , compose and env |
Beta Was this translation helpful? Give feedback.
-
|
Already posted. Log from postgres docker container: Nothing else. Shouldn't break just by updating huh? |
Beta Was this translation helpful? Give feedback.
-
|
Quite odd. However, you can restore the database from the database dump at https://immich.app/docs/administration/backup-and-restore#manual-backup-and-restore |
Beta Was this translation helpful? Give feedback.
-
|
It is also a good opportunity to move your database volume off docker volume and use bind mount instead |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I tried this but seems like it didn't do much, same logs as before. |
Beta Was this translation helpful? Give feedback.
-
|
Have you tried with only one database dump or a few? |
Beta Was this translation helpful? Give feedback.
-
|
Tried with two database dumps from different days. Both didn't work. I've let it sit there for a day with the old dump and now it is up and running again. The only Error now is: |
Beta Was this translation helpful? Give feedback.
-
|
got the same error too after update can't start |
Beta Was this translation helpful? Give feedback.
-
|
Hello, for some data point, is your database on SSD or HDD? |
Beta Was this translation helpful? Give feedback.
-
|
hdd |
Beta Was this translation helpful? Give feedback.
-
|
Got the same issue. Reverting back fixed the problem. For it took down the pg server everytime immich started to connect to it. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, for some data point, is your database on SSD or HDD? |
Beta Was this translation helpful? Give feedback.
-
|
On HDD's, via a iscsi pvc |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, we will take a look into this |
Beta Was this translation helpful? Give feedback.
-
|
Hey Yall, Had the same issue running on kubernetes I had to massively increase the amount of memory the pod postgres was allowed to use and then the application started fine. It seems to be something weird with geocode migrations? My postgres pod is assigned to an hdd server so that could also contribute. |
Beta Was this translation helpful? Give feedback.
-
|
@Eernie how did you overwrite the limits in the helm chart? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I did the following: postgres:
primary:
resources:
requests:
cpu: 500m
memory: 2048Mi
limits:
cpu: 750m
memory: 2048MiThis seems enough for my database and the amount of media immich holds |
Beta Was this translation helpful? Give feedback.
-
|
thanks a lot. this works for me. I think everybody using the kubernetes helm chart will have the issue with too less ram and then postgres will be killed while running the migration. |
Beta Was this translation helpful? Give feedback.
-
|
I made a pull request to add the resource limit for memory: |
Beta Was this translation helpful? Give feedback.
-
|
Worked for me too, thx @Eernie |
Beta Was this translation helpful? Give feedback.
-
|
Ran into this as well starting when going directly to the 1.122.1 release. I shutdown all containers, reverted to the last stable release, restored the DB, and rebooted. Env Images
Steps (only using Postgres and Immich server; ML pod was left scaled down to 0)
Postgres logs: Immich logs |
Beta Was this translation helpful? Give feedback.
-
|
Really strange how the geocoding importing process could have corrupted the database. @zackpollard some more data point here |
Beta Was this translation helpful? Give feedback.
-
|
I think it is the same memory issue, see: #14499 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, increase postgres memory, that should fix it, at least it did for me. |
Beta Was this translation helpful? Give feedback.
-
|
Error: getaddrinfo EAI_AGAIN database Same problem here, the server won't start since the update from 6-12-2024 |
Beta Was this translation helpful? Give feedback.
-
|
Anyone whose logs are saying Anybody else, you have a different problem so please create a new thread to keep this one focused. |
Beta Was this translation helpful? Give feedback.
-
|
This is the full logs, it keeps crashing, here you see 2 'runs': I fixed it by giving postgres more memory as suggested here: #14499 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
I had the same issue after updating to 1.222.2 today: Postgres: 2024-12-11 13:55:36.663 UTC [1] LOG: starting PostgreSQL 16.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit PostgreSQL Database directory appears to contain a database; Skipping initialization 2024-12-11 22:08:12.224 UTC [1] LOG: starting PostgreSQL 16.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit |
Beta Was this translation helpful? Give feedback.
-
|
Nevermind, restarting postgres with the immich containers stopped then restarting immich containers solved it |
Beta Was this translation helpful? Give feedback.
-
|
Also receiving error PostgreSQL Database directory appears to contain a database; Skipping initialization after upgrading to 1.222.0 |
Beta Was this translation helpful? Give feedback.
-
|
That’s not an error, it’s normal |
Beta Was this translation helpful? Give feedback.
-
|
Increasing the postgresql allocated memory (kubernetes in my case) solved this for me as well. Thank you! |
Beta Was this translation helpful? Give feedback.
-
The bug
immich-server container keeps restarting with the attached error log after updating to 1.222.0.
The OS that Immich Server is running on
Ubuntu 24.04
Version of Immich Server
1.222.0
Version of Immich Mobile App
n/a
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
...
Relevant log output
[Nest] 16 - 12/05/2024, 7:13:29 PM LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.122.0] [production] [Nest] 7 - 12/05/2024, 7:13:33 PM ERROR [Microservices:MetadataService] Unable to initialize reverse geocoding: QueryFailedError: Connection terminated unexpectedly QueryFailedError: Connection terminated unexpectedly at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async /usr/src/app/dist/repositories/database.repository.js:201:21 error: the database system is in recovery mode at Parser.parseErrorMessage (/usr/src/app/node_modules/pg-protocol/dist/parser.js:283:98) at Parser.handlePacket (/usr/src/app/node_modules/pg-protocol/dist/parser.js:122:29) at Parser.parse (/usr/src/app/node_modules/pg-protocol/dist/parser.js:35:38) at Socket.<anonymous> (/usr/src/app/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { length: 109, severity: 'FATAL', code: '57P03', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'postmaster.c', line: '2357', routine: 'ProcessStartupPacket' } microservices worker error: error: the database system is in recovery mode microservices worker exited with code 1 Killing api process Initializing Immich v1.122.0 Detected CPU Cores: 10 Starting api worker Starting microservices workerAdditional information
No response
Beta Was this translation helpful? Give feedback.
All reactions