Missing lock for job #14501

almottier · 2024-12-05T13:35:11Z

almottier
Dec 5, 2024

The bug

Hello,

Every time I start the Immich Docker stack, I encounter multiple errors. Additionally, thumbnails are not being generated even when I start the job manually. I have to restart the stack manually to fix the thumbnails issues but still having "Missing lock for job" erros.

Error: Missing lock for job 14. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:17:37 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:17:37 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')

The OS that Immich Server is running on

GNU/Linux synology_apollolake_218+

Version of Immich Server

v1.121.0

Version of Immich Mobile App

?

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      # start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/volume1/immich
# The location where your database files are stored
DB_DATA_LOCATION=/volume1/docker/immich/database

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=xxxxx

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=immich
DB_DATABASE_NAME=immich

Reproduction steps

docker-compose up -d

Relevant log output

sh-4.4# docker logs -f immich_server
Initializing Immich v1.121.0
Detected CPU Cores: 2
Starting api worker
Starting microservices worker
...
Error: Missing lock for job 1. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Error: Missing lock for job 1. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:16:38 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:38 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] Object:
{}

[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:38 PM   ERROR [Microservices:JobService] Object:
{}

Error: Missing lock for job 2. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Error: Missing lock for job 2. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:16:41 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:41 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] Object:
{}

[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:41 PM   ERROR [Microservices:JobService] Object:
{}

Error: Missing lock for job 3. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Error: Missing lock for job 3. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:16:43 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] Object:
{}

[Nest] 7  - 12/05/2024, 1:16:43 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:43 PM   ERROR [Microservices:JobService] Object:
{}

Error: Missing lock for job 4. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Error: Missing lock for job 4. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:16:46 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:46 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] Object:
{}

[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:46 PM   ERROR [Microservices:JobService] Object:
{}

Error: Missing lock for job 5. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Error: Missing lock for job 5. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 34  - 12/05/2024, 1:16:48 PM     LOG [Api:EventRepository] Websocket Disconnect: dMMBThedAyCEYNA_AAAB
[Nest] 34  - 12/05/2024, 1:16:50 PM     LOG [Api:EventRepository] Websocket Connect:    4YWYR-61wiDOHnhfAAAF
[Nest] 7  - 12/05/2024, 1:16:52 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:52 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 7  - 12/05/2024, 1:16:52 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:16:52 PM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:152:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:122:13)
[Nest] 7  - 12/05/2024, 1:16:52 PM   ERROR [Microservices:JobService] Object:
{}

Additional information

No response

Answered by almottier

Dec 12, 2024

I found the source of the problem. It was due to a redis being used by an other service ...

  broker:
    image: redis:6.0
    container_name: redis # renamed to paperless-redis to fix...
    restart: unless-stopped

I thought it was the service name and not the container name that was being used...

You were on the right track @bo0tzz , I had omitted the containers outside of immich from the docker ps -a... I shouldn't have...

Sorry for the noise @alextran1502 ... Immich is great!

View full answer

bo0tzz · 2024-12-05T13:51:13Z

bo0tzz
Dec 5, 2024
Maintainer

How long have you been running Immich for? Can you post the output of docker ps -a to see if there's an old microservices container still around?

0 replies

alextran1502 · 2024-12-05T15:03:07Z

alextran1502
Dec 5, 2024
Maintainer

is this location /volume1/docker/immich/database on an SSD or HDD?

0 replies

almottier · 2024-12-05T18:18:40Z

almottier
Dec 5, 2024
Author

@bo0tzz I don't see any old containers here.

sh-4.4# docker ps -a
d61e71289319   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   5 hours ago     Up 5 hours (healthy)   0.0.0.0:2283->2283/tcp, :::2283->2283/tcp                                                                                                 immich_server
9c3a157b17ae   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.s…"   5 hours ago     Up 5 hours (healthy)   5432/tcp                                                                                                                                  immich_postgres
f4d5544a457d   redis:6.2-alpine                                     "docker-entrypoint.s…"   5 hours ago     Up 5 hours (healthy)   6379/tcp                                                                                                                                  immich_redis
71228076c23d   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     5 hours ago     Up 5 hours (healthy)                                                                                                                                             immich_machine_learning

@alextran1502 it is on an HDD WD Red 5400 RPM.

1 reply

almottier Dec 12, 2024
Author

I found the source of the problem. It was due to a redis being used by an other service ...

  broker:
    image: redis:6.0
    container_name: redis # renamed to paperless-redis to fix...
    restart: unless-stopped

I thought it was the service name and not the container name that was being used...

You were on the right track @bo0tzz , I had omitted the containers outside of immich from the docker ps -a... I shouldn't have...

Sorry for the noise @alextran1502 ... Immich is great!

Answer selected by almottier

alextran1502 · 2024-12-05T18:48:40Z

alextran1502
Dec 5, 2024
Maintainer

Yeah, having a database on HDD will cause that issue since the I/O speed of HDD might not be fast enough for the database's read/write requirement. The database should be running on an SSD

0 replies

almottier · 2024-12-06T04:14:46Z

almottier
Dec 6, 2024
Author

I am using a Synology NAS with 2 bays, containing 2 HDDs. I'm surprised this setup is causing so many issues, as I would expect many people to be running the same configuration. I plan to tweak the PostgreSQL configuration like this do you think that might help?

    command:
      [
        'postgres',
        '-c', 'shared_preload_libraries=vectors.so',
        '-c', 'search_path="$$user", public, vectors',
        '-c', 'logging_collector=on',
        # Reduce from 2GB to 1GB to decrease write pressure
        '-c', 'max_wal_size=1GB',
        # Reduce from 512MB to 256MB
        '-c', 'shared_buffers=256MB',
        '-c', 'wal_compression=on',
        # New optimizations
        '-c', 'random_page_cost=4.0',
        '-c', 'effective_io_concurrency=2',
        '-c', 'checkpoint_completion_target=0.9',
        '-c', 'synchronous_commit=off'
      ]

0 replies

almottier · 2024-12-06T04:22:42Z

almottier
Dec 6, 2024
Author

Even with this configuration, just by uploading a single photo from my Android device, I immediately get a new error:

[Nest] 6  - 12/06/2024, 4:18:02 AM   ERROR [Microservices:JobService] Unable to run job handler (metadataExtraction/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')
[Nest] 6  - 12/06/2024, 4:18:02 AM   ERROR [Microservices:JobService] TypeError: Cannot read properties of undefined (reading 'replaceAll')
    at JobService.onJobStart (/usr/src/app/dist/services/job.service.js:149:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
[Nest] 6  - 12/06/2024, 4:18:02 AM   ERROR [Microservices:JobService] Object:
{}

Error: Missing lock for job 1. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)

Are you sure the problem is related to HDD vs SSD?

10 replies

almottier Dec 6, 2024
Author

Sure.

sh-4.4# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda               8:0    0  2.7T  0 disk
├─sda1            8:1    0  2.4G  0 part
│ └─md0           9:0    0  2.4G  0 raid1 /
├─sda2            8:2    0    2G  0 part
│ └─md1           9:1    0    2G  0 raid1 [SWAP]
└─sda5            8:5    0  2.7T  0 part
  └─md2           9:2    0  2.7T  0 raid1
    └─vg1000-lv 248:0    0  2.7T  0 lvm   /volume1/@docker
                                          /volume1
sdb               8:16   0  2.7T  0 disk
├─sdb1            8:17   0  2.4G  0 part
│ └─md0           9:0    0  2.4G  0 raid1 /
├─sdb2            8:18   0    2G  0 part
│ └─md1           9:1    0    2G  0 raid1 [SWAP]
└─sdb5            8:21   0  2.7T  0 part
  └─md2           9:2    0  2.7T  0 raid1
    └─vg1000-lv 248:0    0  2.7T  0 lvm   /volume1/@docker
                                          /volume1
sdq              65:0    0  5.5T  0 disk
└─sdq1           65:1    0  5.5T  0 part  /volumeUSB1/usbshare
synoboot        135:240  0  120M  0 disk
├─synoboot1     135:241  0   32M  0 part
└─synoboot2     135:242  0   84M  0 part
zram0           249:0    0  2.9G  0 disk  [SWAP]
zram1           249:1    0  2.9G  0 disk  [SWAP]

sh-4.4# docker exec immich_server lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0  2.7T  0 disk
|-sda1        8:1    0  2.4G  0 part
| `-md0       9:0    0  2.4G  0 raid1 /usr/share/zoneinfo/Etc/UTC
|-sda2        8:2    0    2G  0 part
| `-md1       9:1    0    2G  0 raid1 [SWAP]
`-sda5        8:5    0  2.7T  0 part
  `-md2       9:2    0  2.7T  0 raid1
sdb           8:16   0  2.7T  0 disk
|-sdb1        8:17   0  2.4G  0 part
| `-md0       9:0    0  2.4G  0 raid1 /usr/share/zoneinfo/Etc/UTC
|-sdb2        8:18   0    2G  0 part
| `-md1       9:1    0    2G  0 raid1 [SWAP]
`-sdb5        8:21   0  2.7T  0 part
  `-md2       9:2    0  2.7T  0 raid1
sdq          65:0    0  5.5T  0 disk
`-sdq1       65:1    0  5.5T  0 part
synoboot    135:240  0  120M  0 disk
|-synoboot1 135:241  0   32M  0 part
`-synoboot2 135:242  0   84M  0 part
zram0       249:0    0  2.9G  0 disk  [SWAP]
zram1       249:1    0  2.9G  0 disk  [SWAP]

alextran1502 Dec 6, 2024
Maintainer

Looks good, can you try update to v1.121.1 that just released? We fixed some issues with thumbnail image cannot be displayed

almottier Dec 6, 2024
Author

I confirm v.1.122.1 fixed the thumbnails issue 👍 , but I am still facing multiples Error: Missing lock for job x. failed errors.

alextran1502 Dec 6, 2024
Maintainer

Perhaps try restoring a new database to a different location on your SSD OS drive, see if it fixes the issue

almottier Dec 6, 2024
Author

My Synology NAS doesn't have an SSD installed, OS is on the HDD. The only workaround I've found is to set up a crontab that restarts the Immich stack daily...

samcre · 2024-12-11T10:11:11Z

samcre
Dec 11, 2024

I'm facing the same problem. I'm running the latest versions (1.222.2) and my database storage it's on a SSD. Is there anything I can check?

4 replies

samcre Dec 11, 2024

Funny enough, running v1.220.2 makes thumbnail generation works perfectly. 🤔

coltenkrauter Dec 14, 2024

I too am seeing these errors... When folks talk about moving database data to an SSD, are they talking about Postgres data? I assume so, but in the error message,

Error: Missing lock for job 14. failed
    at Scripts.finishedErrors (/usr/src/app/node_modules/bullmq/dist/cjs/classes/scripts.js:272:24)
    at Job.moveToFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/job.js:427:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async handleFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:379:21)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 12/05/2024, 1:17:37 PM    WARN [Microservices:JobRepository] Skipping unknown job: "undefined"
[Nest] 7  - 12/05/2024, 1:17:37 PM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/undefined): TypeError: Cannot read properties of undefined (reading 'replaceAll')

I see it references bullmq, isn't that a Redis thing? But, a lockfile sounds like a Postgres thing.

BullMQ is a Node.js library that implements a fast and robust queue system built on top of Redis that helps in resolving many modern age

I would like to clarify which data needs to be moved to an SSD... Redis data? Or Postgres data?

bo0tzz Dec 15, 2024
Maintainer

When people talk about putting the database on an SSD that's almost always Postgres. This error is entirely Redis though.

Like upthread, are you running any other applications that use Redis and might accidentally be connecting to this instance?

coltenkrauter Dec 15, 2024

No I don't see any other applications that could be connecting to this instance..., I restarted my stack, and I am no longer seeing the lock file error. There is another redis error, but I'll move to a different discussion or issue for that. Ty

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing lock for job #14501

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Missing lock for job #14501

almottier Dec 5, 2024

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Relevant log output

Additional information

Replies: 7 comments · 15 replies

bo0tzz Dec 5, 2024 Maintainer

alextran1502 Dec 5, 2024 Maintainer

almottier Dec 5, 2024 Author

almottier Dec 12, 2024 Author

alextran1502 Dec 5, 2024 Maintainer

almottier Dec 6, 2024 Author

almottier Dec 6, 2024 Author

almottier Dec 6, 2024 Author

alextran1502 Dec 6, 2024 Maintainer

almottier Dec 6, 2024 Author

alextran1502 Dec 6, 2024 Maintainer

almottier Dec 6, 2024 Author

samcre Dec 11, 2024

samcre Dec 11, 2024

coltenkrauter Dec 14, 2024

bo0tzz Dec 15, 2024 Maintainer

coltenkrauter Dec 15, 2024

almottier
Dec 5, 2024

Replies: 7 comments 15 replies

bo0tzz
Dec 5, 2024
Maintainer

alextran1502
Dec 5, 2024
Maintainer

almottier
Dec 5, 2024
Author

almottier Dec 12, 2024
Author

alextran1502
Dec 5, 2024
Maintainer

almottier
Dec 6, 2024
Author

almottier
Dec 6, 2024
Author

almottier Dec 6, 2024
Author

alextran1502 Dec 6, 2024
Maintainer

almottier Dec 6, 2024
Author

alextran1502 Dec 6, 2024
Maintainer

almottier Dec 6, 2024
Author

samcre
Dec 11, 2024

bo0tzz Dec 15, 2024
Maintainer