Podman with Nexus proxy - unsupported schema version 2 #14447

ChrisitanAkerblom · 2022-06-01T14:21:30Z

ChrisitanAkerblom
Jun 1, 2022

I'm using podman 4.0.2 in an offgrid environment with a Nexus proxy behind an Artifactory. The Artifactory proxy is the only way to retrieve images from. Artifactory has no long term storage guaranteed for the images so I need the Nexus to store the downloaded images long term.

When I try to pull images I get the following error when going through the Nexus proxy, but not when only going through the Artifactory proxy:

> podman pull hello-world
Resolved "hello-world" as an alias (/home/xxxxx/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/hello-world:latest...
Error: initializing image from source docker://hello-world:latest: unsupported schema version 2

Nexus contains json files for each image downloaded, latests and a sha256:xxxxx. The content in each of the json files contains:
"mediaType": "application\/vnd.docker.distribution.manifest.list.v2+json"

Can anyone give me some hints on how to fix this or further investigate why I get this error?

mheon · 2022-06-01T15:47:35Z

mheon
Jun 1, 2022
Maintainer

@vrothberg PTAL

0 replies

ChrisitanAkerblom · 2022-06-21T13:07:00Z

ChrisitanAkerblom
Jun 21, 2022
Author

This is the output I get with --log-level=debug:

podman pull --log-level=debug hello-world
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug hello-world) 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/xxxxx/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/xxxxx/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/nnnnnnnnn/containers 
DEBU[0000] Using static dir /home/xxxxx/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/nnnnnnnnn/libpod/tmp 
DEBU[0000] Using volume path /home/xxxxx/.local/share/containers/storage/volumes 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/runc"            
INFO[0000] Setting parallel job count to 7              
DEBU[0000] Pulling image hello-world (policy: always)   
DEBU[0000] Looking up image "hello-world" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "hello-world" ...                     
DEBU[0000] Loading registries configuration "/home/xxxxx/.config/containers/registries.conf" 
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/xxxxx/.local/share/containers/storage+/run/user/nnnnnnnnn/containers]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage 
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/home/xxxxx/.local/share/containers/storage+/run/user/nnnnnnnnn/containers]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412) 
DEBU[0000] Image hello-world resolved to local image docker.io/library/hello-world:latest which will be used for pulling 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/hello-world:latest for docker.io/library/hello-world:latest 
DEBU[0000] parsed reference into "[overlay@/home/xxxxx/.local/share/containers/storage+/run/user/nnnnnnnnn/containers]docker.io/library/hello-world:latest" 
Trying to pull docker.io/library/hello-world:latest...
DEBU[0000] Copying source image //hello-world:latest to destination image [overlay@/home/xxxxx/.local/share/containers/storage+/run/user/nnnnnnnnn/containers]docker.io/library/hello-world:latest 
DEBU[0000] Trying to access "the.private.nexus.repository.com/library/hello-world:latest" 
DEBU[0000] No credentials matching the.private.nexus.repository.com/library/hello-world found in /run/user/nnnnnnnnn/containers/auth.json 
DEBU[0000] No credentials matching the.private.nexus.repository.com/library/hello-world found in /home/xxxxx/.config/containers/auth.json 
DEBU[0000] No credentials matching the.private.nexus.repository.com/library/hello-world found in /home/xxxxx/.docker/config.json 
DEBU[0000] No credentials matching the.private.nexus.repository.com/library/hello-world found in /home/xxxxx/.dockercfg 
DEBU[0000] No credentials for the.private.nexus.repository.com/library/hello-world found 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for the.private.nexus.repository.com/library/hello-world:latest, using built-in default file:///home/xxxxx/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/the.private.nexus.repository.com 
DEBU[0000] GET https://the.private.nexus.repository.com/v2/ 
DEBU[0000] Ping https://the.private.nexus.repository.com/v2/ status 401 
DEBU[0000] GET https://the.private.nexus.repository.com/v2/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=https%3A%2F%2Fthe.private.nexus.repository.com%3A8447%2Fv2%2Ftoken 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] GET https://the.private.nexus.repository.com/v2/library/hello-world/manifests/latest 
DEBU[0000] Content-Type from manifest GET is "application/octet-stream" 
DEBU[0000] Using blob info cache at /home/xxxxx/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] Error pulling candidate docker.io/library/hello-world:latest: initializing image from source docker://hello-world:latest: unsupported schema version 2 
Error: initializing image from source docker://hello-world:latest: unsupported schema version 2

The Nexus proxy repository contains two files after this:

The content of the files are:

latest

{
    "manifests": [
        {
            "digest": "sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:7b8b7289d0536a08eabdf71c20246e23f7116641db7e1d278592236ea4dcb30c",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v5"
            },
            "size": 525
        },
        {
            "digest": "sha256:f130bd2d67e6e9280ac6d0a6c83857bfaf70234e8ef4236876eccfbd30973b1c",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            },
            "size": 525
        },
        {
            "digest": "sha256:432f982638b3aefab73cc58ab28f5c16e96fdb504e8c134fc58dff4bae8bf338",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            },
            "size": 525
        },
        {
            "digest": "sha256:995efde2e81b21d1ea7066aa77a59298a62a9e9fbb4b77f36c189774ec9b1089",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "386",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:eb11b1a194ff8e236a01eff392c4e1296a53b0fb4780d8b0382f7996a15d5392",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "mips64le",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:b836bb24a270b9cc935962d8228517fde0f16990e88893d935efcb1b14c0017a",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:98c9722322be649df94780d3fbe594fce7996234b259f27eac9428b84050c849",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "riscv64",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:c7b6944911848ce39b44ed660d95fb54d69bbd531de724c7ce6fc9f743c0b861",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:1e0054d86f2721056c9db5c2322fb8883d185fa1165bca69cb0ac88eeb40be30",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "windows",
                "os.version": "10.0.20348.707"
            },
            "size": 1125
        },
        {
            "digest": "sha256:02a91a3365149788cc22339941adefe48166286642688a1f094c3517da4e37a5",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "windows",
                "os.version": "10.0.17763.2928"
            },
            "size": 1125
        }
    ],
    "mediaType": "application\/vnd.docker.distribution.manifest.list.v2+json",
    "schemaVersion": 2
}

sha256:80f31da1ac7b312ba29d65080fddf797dd76acfb870e677f390d5acba9741b17

{
    "manifests": [
        {
            "digest": "sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:7b8b7289d0536a08eabdf71c20246e23f7116641db7e1d278592236ea4dcb30c",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v5"
            },
            "size": 525
        },
        {
            "digest": "sha256:f130bd2d67e6e9280ac6d0a6c83857bfaf70234e8ef4236876eccfbd30973b1c",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            },
            "size": 525
        },
        {
            "digest": "sha256:432f982638b3aefab73cc58ab28f5c16e96fdb504e8c134fc58dff4bae8bf338",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            },
            "size": 525
        },
        {
            "digest": "sha256:995efde2e81b21d1ea7066aa77a59298a62a9e9fbb4b77f36c189774ec9b1089",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "386",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:eb11b1a194ff8e236a01eff392c4e1296a53b0fb4780d8b0382f7996a15d5392",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "mips64le",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:b836bb24a270b9cc935962d8228517fde0f16990e88893d935efcb1b14c0017a",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:98c9722322be649df94780d3fbe594fce7996234b259f27eac9428b84050c849",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "riscv64",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:c7b6944911848ce39b44ed660d95fb54d69bbd531de724c7ce6fc9f743c0b861",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            },
            "size": 525
        },
        {
            "digest": "sha256:1e0054d86f2721056c9db5c2322fb8883d185fa1165bca69cb0ac88eeb40be30",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "windows",
                "os.version": "10.0.20348.707"
            },
            "size": 1125
        },
        {
            "digest": "sha256:02a91a3365149788cc22339941adefe48166286642688a1f094c3517da4e37a5",
            "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "windows",
                "os.version": "10.0.17763.2928"
            },
            "size": 1125
        }
    ],
    "mediaType": "application\/vnd.docker.distribution.manifest.list.v2+json",
    "schemaVersion": 2
}

Is there anything else I can look into locally for finding more information about the problem at hand?

3 replies

mtrmac Jun 22, 2022
Maintainer

DEBU[0000] Content-Type from manifest GET is "application/octet-stream"

The proxy, or maybe the original registry is providing an invalid value here. It must be fixed to return the right value (this is specific to each individual manifest, it can’t be a simple “everything with JSON suffix, or everything in a /manifests/ URL, should report a single value).

mtrmac Jun 22, 2022
Maintainer

the “invalid value here” refers to a Content-Type header in the HTTP response by the contacted server. (The mediaType value inside the manifest is not the same thing. It might make a difference, I guess, depending on the server’s design, but it’s the server’s responsibility to return a correct value in the HTTP response header as well.)

ChrisitanAkerblom Jun 23, 2022
Author

I'll look into the configuration of the proxy repository in the Nexus server. There are not that much to configure though but I guess it could be a bug in Nexus.

ChrisitanAkerblom · 2022-06-22T06:32:48Z

ChrisitanAkerblom
Jun 22, 2022
Author

It is possible to workaround this problem with the following procedure:

Pull image without going through the Nexus proxy repository.
Push image to a local Nexus repository.
From this point on, pull the image from the local Nexus repository.

I'm not sure that this is a problem with podman, but it would be good to continue the investigation further to find the root cause to the problem.

4 replies

rhatdan Jun 22, 2022
Maintainer

@mtrmac PTAL

mtrmac Jun 22, 2022
Maintainer

That workaround seems unlikely. Are steps 1. and 3. both done from the same computer? If so, does that work if the local image store is completely purged in between?

How do the two images actually differ (maybe using skopeo inspect --raw docker://… to compare)?

ChrisitanAkerblom Jun 23, 2022
Author

All three steps are performed on the same computer. The local image store is completely purged between step 2 and 3.

I do not have access to skopeo in the environment so I cannot inspect the images. What I can see in Nexus is however that the manifest and tag files differs. I've attached the non proxy file contents below. They are completely different from the files downloaded when pulling the image through the Nexus proxy repository.

latest

{
    "schemaVersion": 1,
    "name": "hello-world",
    "tag": "latest",
    "architecture": "amd64",
    "fsLayers": [
        {
            "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
        },
        {
            "blobSum": "sha256:44f2b372045fd4cf40c7bbc586b2971b4c9960d61b4de9db9afea9e61f5b34ce"
        }
    ],
    "history": [
        {
            "v1Compatibility": "{\"created\":\"2021-09-23T23:47:57.442225064Z\",\"docker_version\":\"20.10.7\",\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/hello\"],\"Image\":\"sha256:b9935d4e8431fb1a7f0989304ec86b3329a99a25f5efdc7f09f3f8c41434ca6d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"8746661ca3c2f215da94e6d3f7dfdcafaff5ec0b21c9aff6af3dc379a82fbc72\",\"os\":\"linux\",\"container_config\":{\"Hostname\":\"8746661ca3c2\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"/hello\\\"]\"],\"Image\":\"sha256:b9935d4e8431fb1a7f0989304ec86b3329a99a25f5efdc7f09f3f8c41434ca6d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"id\":\"9b10e7ff0cce1788e0081a0542b7cc59e50e85326baf344695b56de9e1554a85\",\"parent\":\"8127ee591a02dbc3d2d46d698f40a60072337f225776398aa14cdf1b49e97327\"}"
        },
        {
            "v1Compatibility": "{\"id\":\"8127ee591a02dbc3d2d46d698f40a60072337f225776398aa14cdf1b49e97327\",\"created\":\"2021-09-23T23:47:57.098990892Z\",\"Cmd\":{\"container_config\":\"/bin/sh -c #(nop) COPY file:50563a97010fd7ce1ceebd1fa4f4891ac3decdf428333fb2683696f4358af6c2 in / \"}}"
        }
    ]
}

sha256...

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1469,
        "digest": "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 2552,
            "digest": "sha256:44f2b372045fd4cf40c7bbc586b2971b4c9960d61b4de9db9afea9e61f5b34ce"
        }
    ]
}

I'm going on vacation for 4 weeks so I will not be able to respond until 25th of July.

mtrmac Jun 23, 2022
Maintainer

That… mostly seems to just introduce extra variables into the investigation.

The core of the issue is that with invalid Content-Type, the code defaults to the oldest schema1 ("schemaVersion": "1"). Something in the workaround process, for a probably fairly independent reason, triggers conversion of the original multi-arch image (80f31da1ac7b312ba29d65080fddf797dd76acfb870e677f390d5acba9741b17) to schema1; probably pushes using other versions are rejected, for some reason, and the push code fall backs to schema1. (Or, alternatively, the proxy is so badly misconfigured, that it triggers a server-side conversion to schema1.) Either way, now that the freshly-pushed image really is schema1, it probably matches with the invalid Content-type header, and can be pulled successfully.

(Also note that the workaround, using podman pull + podman push, discards all per-architecture variants except for the original one.)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman with Nexus proxy - unsupported schema version 2 #14447

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Podman with Nexus proxy - unsupported schema version 2 #14447

ChrisitanAkerblom Jun 1, 2022

Replies: 3 comments · 7 replies

mheon Jun 1, 2022 Maintainer

ChrisitanAkerblom Jun 21, 2022 Author

mtrmac Jun 22, 2022 Maintainer

mtrmac Jun 22, 2022 Maintainer

ChrisitanAkerblom Jun 23, 2022 Author

ChrisitanAkerblom Jun 22, 2022 Author

rhatdan Jun 22, 2022 Maintainer

mtrmac Jun 22, 2022 Maintainer

ChrisitanAkerblom Jun 23, 2022 Author

mtrmac Jun 23, 2022 Maintainer

ChrisitanAkerblom
Jun 1, 2022

Replies: 3 comments 7 replies

mheon
Jun 1, 2022
Maintainer

ChrisitanAkerblom
Jun 21, 2022
Author

mtrmac Jun 22, 2022
Maintainer

mtrmac Jun 22, 2022
Maintainer

ChrisitanAkerblom Jun 23, 2022
Author

ChrisitanAkerblom
Jun 22, 2022
Author

rhatdan Jun 22, 2022
Maintainer

mtrmac Jun 22, 2022
Maintainer

ChrisitanAkerblom Jun 23, 2022
Author

mtrmac Jun 23, 2022
Maintainer