Photos (HEIC) are corrupted when uploading via iOS app

[obale]

The bug

Photos (HEIC) uploaded via the Immich iOS app are showing up as corrupted in the Immich web interface and as (corrupted) replicated image on mobile. The corrupted photos are visualized flawed and are showing green (anod other colored) artifacts. These artifacts are not always in the same location when deleting and re-uploading the photo, at least as far as I can tell.

In version v1.121.0 and before I didn't experience this issue. Downgrading the server to v1.121.0 didn't solve the issue, so it may be related to the iOS version or a combination of both (server and mobile).

Was searching for a related issue, but couldn't find any.

The OS that Immich Server is running on

Ubuntu 24.04.1 LTS

Version of Immich Server

v1.122.1

Version of Immich Mobile App

v1.122.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#                        
# WARNING: Make sure to use the docker-compose.yml of the current release:
#              
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#                                           
# The compose file on main may not be compatible with the latest release.
#                            
                                                                       
name: immich                 
                                                                       
services:          
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:  
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /mnt/nextcloud:/mnt/nextcloud:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=[CENSORED}


# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=pgdata

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Open iOS mobile app and upload a photo
2. Check the web interface for the photo. The photo is either showing up with (green) artifacts, indicating that the file is corrupted or no thumbnail is generated (not sure if that is unrelated and fixed by now).
3. Check the iOS app to see the uploaded photo being replicated, one version is the original and one is the corrupted one.

Relevant log output

immich_server            | [Nest] 18  - 12/08/2024, 9:28:04 PM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.122.1] [production] 
immich_server            | [Nest] 18  - 12/08/2024, 9:28:17 PM     LOG [Api:EventRepository] Websocket Connect:    M_FX6JytsMXxu2DzAAAB
immich_server            | [Nest] 18  - 12/08/2024, 9:28:19 PM     LOG [Api:EventRepository] Websocket Disconnect: M_FX6JytsMXxu2DzAAAB
immich_server            | [Nest] 18  - 12/08/2024, 9:28:20 PM     LOG [Api:EventRepository] Websocket Connect:    4QtHIi_eTn3YIYZ1AAAD
immich_server            | [Nest] 18  - 12/08/2024, 9:28:30 PM     LOG [Api:EventRepository] Websocket Connect:    mPXfSQvIA_W4POYGAAAF       
immich_server            | [Nest] 18  - 12/08/2024, 9:28:32 PM     LOG [Api:EventRepository] Websocket Connect:    9H78SuAWjaXxJ6-YAAAH
immich_machine_learning  | [12/08/24 21:28:39] INFO     Loading detection model 'buffalo_l' to memory                                         
immich_machine_learning  | [12/08/24 21:28:39] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of                       
immich_machine_learning  |                              preference                                          
immich_machine_learning  | [12/08/24 21:28:39] INFO     Loading visual model 'ViT-B-32__openai' to memory  
immich_machine_learning  | [12/08/24 21:28:39] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                          
immich_machine_learning  | [12/08/24 21:28:42] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [12/08/24 21:28:42] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                          
immich_server            | [Nest] 18  - 12/08/2024, 9:28:59 PM     LOG [Api:EventRepository] Websocket Disconnect: 9H78SuAWjaXxJ6-YAAAH
immich_server            | [Nest] 18  - 12/08/2024, 9:29:22 PM     LOG [Api:EventRepository] Websocket Disconnect: 4QtHIi_eTn3YIYZ1AAAD
immich_server            | [Nest] 18  - 12/08/2024, 9:29:23 PM     LOG [Api:EventRepository] Websocket Connect:    4fqbhA9d9TETaplXAAAJ
immich_server            | [Nest] 18  - 12/08/2024, 9:29:40 PM     LOG [Api:EventRepository] Websocket Disconnect: mPXfSQvIA_W4POYGAAAF

Additional information

No response

[alextran1502]

Do you mind including photos shown as corrupted here?

[alextran1502]

Also, please try to bring your instance down, remove the images, repull and then bring the instance up and try uploading again

[obale]

I removed in the past multiple times the containers (via docker compose down), now I also removed all the images via docker rmi -f $(docker images -aq). For good measure I was also running docker system prune. After starting immich via docker compose up -d . Same problem, no luck.

The photos have some sensitive meta data in them, such as location data, but I will see what I can do. In the meantime the screenshot below shows the artifact issue.

Screenshot From 2024-12-09 12-51-59

Some of the maybe relevant meta data is:

IMG_1298.HEIC
upload/library/obale/2024/2024-12-08/IMG_1298+2.HEIC

9 MP 2268 x 4032 
1.2 MiB

Apple iPhone 11 Pro
ƒ/1.8 1/60 s 4.25 mm ISO 200

I have a pretty standard install on an Ubuntu VM in Proxmox, so I hoped someone had experienced the same issue.

[alextran1502]

If you want, you can send the photo directly from the Photos app to my email at [email protected]

[obale]

I shot a completely new photo and remove the location from the iOS Camera app (send this you, @alextran1502, via email). Some interesting observations:

• It corrupts the image after upload, as outlined in this ticket.
• It adds back the location to the meta data, that is why I shot a completely new photo to assure there is no overlap with the file name or image content hash. Potentially it derives the information from other meta data fields.

Hope that helps.

[obale]

I am wondering if I can help debug this issue better by understanding when and how the uploaded photos are modified. Can someone point me to the right direction to look into that or provide some context how the original files are modified after or during successful uploads? Thank you!

[obale]

Not conclusive, because of small sample size, but it looks like the images corrupted are from the local Camera (latest iOS and immich app version). Photos shared seem to upload just fine.

[alextran1502]

Uploaded the file. This is how it looks from my instance.

It doesn't look corrupted from the screenshot, correct?

What is your camera settings?

[obale]

It looks not corrupted and if it would be it would show up as duplicated in the Immich app (I guess at least there is a hash check somewhere to detect corruption).

My camera settings are as follows:

• Formats: High Efficiency (HEIC)
• Smart HDR: on
• Location data is also on
• System language is English (hence no special characters)

@alextran1502, how are images modified during upload?

[alextran1502]

Image is not modified during upload 🤔

[alextran1502]

Do you mind try turn the phone off and on. Then try uploading a newly taken photo?

[obale]

That is strange that they are not modified, but somehow corrupted consistently and randomly. I restarted my phone multiple times, same result. I even removed the immich app and installed it again.

[alextran1502]

Perhaps you can try creating a new Immich instance with brand new docker images and try uploading photo to that instance

[akostadinov]

@alextran1502 , there might be some issue with upload handling because I see corruption uploading via web. See #14657

There is a link to a test file. It might be related to server being underpowered or something. Given client is doing the hashing, I wonder if server checks the hash after file being uploaded.

Update: actually for me the problem turns out to be the download, not upload because I see on server the file has the correct hash. Would you check if the files on the server are correct in your case or have been uploaded broken?

[obale]

Seems to be corrupted during upload, because when I am downloading via ssh (scp) and are opening local I see the same artifacts. Didn't compare the hash against the original, given that it is obviously corrupted and it can be visually verified.

Trying now to upgrade to the latest version published today.

[akostadinov]

@obale then also good to check whether images downloaded from phone directly open properly. Maybe the format is not good for your PC?

[alextran1502]

@obale Are you using iCloud-optimized storage?

[obale]

The problem may be the AAE format. It stores the images in JPG and AAE format. As far as I can tell it uploads AAE. I am checking now if I am using iCloud optimized storage, but I am not backing up the photos to iCloud.

[alextran1502]

How do you set your phone to capture the photo in those formats?

[obale]

Under Settings -> Camera -> Formats

[obale]

Even in JPG format they are corrupted, not always, but most times.

[obale]

Just as reference and because resources have been mentioned. I am running immich in a Proxmox VM (Ubuntu 24.04.1 LTS) with 24 GB of memory and 15 processors (2 Sockets and 8 Cores). Nothing else is running in the VM.

[alextran1502]

What was is the format you have selected? Can you share the screenshot?

[obale]

I tried both formats.

[alextran1502]

Thanks. I can't reproduce this issue with all format. We will need to be able to reproduce this for a fix.

Can you try create a new test Immich instance with all stock settings and try uploading some photos there?

[obale]

Will do that.