Digest inconsistent for multi-arch images id vs name.

[traylenator]

Issue Description

The digest of an image used to be consistent when queried from a repository but for some (at least) multi-arch images this is no longer the case.

 podman image inspect 8e75cbc5b25c  | jq .[].Digest
"sha256:02ffd439b71d9ea9408e449b568f65c0bbbb94bebd8750f1d80231ab6496008e"

however

$ podman image inspect docker.io/library/nginx:1.23-alpine  | jq .[].Digest
"sha256:01ccf4035840dd6c25042b2b5f6b09dd265b4ed5aa7b93ccc4714027c0ce5685"

So by id or by name are different and in particular do not match unique Digest in the repository:

$ skopeo inspect docker://docker.io/nginx:1.23-alpine | jq .Digest
"sha256:02ffd439b71d9ea9408e449b568f65c0bbbb94bebd8750f1d80231ab6496008e"

So the unique digest is inconsistent from the choice of Digests.

podman image inspect docker.io/library/nginx:1.23-alpine  | jq .[].RepoDigests
[
  "docker.io/library/nginx@sha256:01ccf4035840dd6c25042b2b5f6b09dd265b4ed5aa7b93ccc4714027c0ce5685",
  "docker.io/library/nginx@sha256:02ffd439b71d9ea9408e449b568f65c0bbbb94bebd8750f1d80231ab6496008e"
]

Steps to reproduce the issue

Steps to reproduce the issue

1. $ skopeo inspect docker://docker.io/nginx:1.23-alpine | jq .Digest

"sha256:02ffd439b71d9ea9408e449b568f65c0bbbb94bebd8750f1d80231ab6496008e"

2. podman pull docker://docker.io/nginx:1.23-alpine
3. podman image list

REPOSITORY               TAG          IMAGE ID      CREATED        SIZE
docker.io/library/nginx  1.23-alpine  8e75cbc5b25c  20 months ago  42.8 MB

4. podman image inspect docker.io/library/nginx:1.23-alpine | jq .[].Digest
5. podman image inspect 8e75cbc5b25c | jq .[].Digest

The two digest from the last commands do not match.

Describe the results you received

Describe the results you received

$ podman image inspect docker.io/library/nginx:1.23-alpine  | jq .[].Digest
"sha256:01ccf4035840dd6c25042b2b5f6b09dd265b4ed5aa7b93ccc4714027c0ce5685"
$ podman image inspect 8e75cbc5b25c | jq .[].Digest
"sha256:02ffd439b71d9ea9408e449b568f65c0bbbb94bebd8750f1d80231ab6496008e"

digests do not match.

Describe the results you expected

Describe the results you expected

• podman image inspect docker.io/library/nginx:1.23-alpine | jq .[].Digest
• podman image inspect 8e75cbc5b25c | jq .[].Digest
• skopeo inspect docker://docker.io/nginx:1.23-alpine | jq .Digest

should produce a consistent Digest after an image is pulled.

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 98.49
    systemPercent: 0.59
    userPercent: 0.92
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.13.0-0.rc2.20241211gitf92f4749861b.24.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 80388096
  memTotal: 2048278528
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.19
      commit: db31c42ac46e20b5527f5339dcbf6f023fcd539c
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241211.g09478d5-1.fc42.x86_64
    version: |
      pasta 0^20241211.g09478d5-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 1029328896
  swapTotal: 2047864832
  uptime: 0h 6m 25.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/steve/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/steve/.local/share/containers/storage
  graphRootAllocated: 20397948928
  graphRootUsed: 4660064256
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/steve/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732665600
  BuiltTime: Wed Nov 27 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

This is latest release in Fedora rawhide.

Same change in behaviour also occurred between podman 4 and podman 5 with RHEL 9.4->9.5.
https://issues.redhat.com/browse/RHEL-68539

Additional information

Probably only with multi-arch images.

[mtrmac]

There is a much longer explanation internally in https://issues.redhat.com/browse/RHEL-68539, where this comes from.

For now, I’ll just note that:

Describe the results you expected

• podman image inspect docker.io/library/nginx:1.23-alpine | jq .[].Digest
• podman image inspect 8e75cbc5b25c | jq .[].Digest

is impossible: 8e75cbc5b25c may have many different names, including nginx@sha256:$multiArchDigest and nginx@sha256:perArchDigest and nginx@sha256:$recompressedDifferentDigest. When inspecting by ID, the Digest field only reports one value, and it fundamentally can’t match 2/3 or more.

The caller’s assumptions must be revisited.

Now, the digest tracking in Podman is known to be problematic, but fixing that would not affect the design issue above.