ros:noetic is 3 years old?

[gavanderhoorn]

I apologise in advance for opening this issue -- as I believe it's a topic well-discussed, such as in #671, #112 and #723 -- but after searching old(er) ones I couldn't find something that explained what docker images is telling me.

On my system (and some other systems), I'm getting the following output:

$ docker images ros:noetic
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ros          noetic    d408a6770189   3 years ago   933MB

And indeed:

$ docker inspect -f '{{ .Created }}' ros:noetic
2020-11-17T19:36:01Z

apt update && apt list --upgradable does not mention any Noetic packages as having any updates available though, which I would expect if the image was really three years old.

The same "3 years ago" is shown for images like ros:noetic-ros-base and ros:noetic-robot.

The last one actually aged "visibly" for me, as before I docker pulled it, it was 11 months old (on this particular machine, ID: d63508127062), while after pulling it, it was suddenly 3 years old (ID: 120ea3b74897).

Have I forgotten how to interpret the output of docker images, or is something not as it should be?

[tfoote]

The last published noetic tag is from "a month ago" https://hub.docker.com/layers/library/ros/noetic/images/sha256-c7631b6323509f943142bd494abf268127a6807d81bd75e5f1fec2e0cd2caf7f?context=explore

And if I pull it shows I'm up to date with the long hash version.

$ docker pull ros:noetic
noetic: Pulling from library/ros
Digest: sha256:e3866df3b9798c4f3b49946d61b179745a22409ee8ce7037411ea550581b12b4
Status: Image is up to date for ros:noetic
docker.io/library/ros:noetic
$ docker images | grep noetic
ros                               noetic                              d408a6770189   3 years ago     933MB

That's really odd behavior.

[gavanderhoorn]

After some more creative searching (ie: different keywords), this might be due to a change in the way the official images are built.

I haven't yet found out if something changed there, but according to moby/buildkit/docs/build-repro.md: SOURCE_DATE_EPOCH, it's possible to specify a value which in the end sets the Created field in the image's manifest.

According to this in docker-library/official-images/library/ros, the commit (in this repository) the official images are based on for noetic is df19ab7d5993d3b78a908362cdcd1479a8e78b35.

And it turns out df19ab7 was committed on Tue, 17 Nov 2020 11:36:01 -0800.

Which is exactly what is stored in the Created field of the image (after compensating for timezones).

Apparently the official images now have their created date set to the date of the last commit that 'touched' their Dockerfile.

I guess for images which don't have the problems with updating (as discussed in #112, #671 and #723) this can work/makes sense.

For images like ros:noetic I must say I find it rather confusing.

[gavanderhoorn]

@ruffsl @mikaelarguedas would you know of any changes to how the images are built upstream? Or has this always been how things worked and have I just not been paying attention?

[ruffsl]

Hey @gavanderhoorn , thanks for sleuthing! I'll admit I simallary find it a peculiar choice to set SOURCE_DATE_EPOCH to track the touch date of the source Dockerfile commit. Perhaps as a token monicar to track source file modifications. Here is a tangent ticket from the library repo that discusses the notion of fixating the timestamp deterministically to enable reproducible builds, so perhaps that is a motivation so that rebuilds in time that don't change at a binary level can map to the same exact time hash digest. It's been awhile since I read the thread, so don't recall precicly:

• Reproducible builds docker-library/official-images#16044

That said, instead of relying on this "Created" date, one can also check the Annotations for the open container label org.opencontainers.image.created for a timestamp that more reflects the image's true "creation" time. TBH, I'm not sure how Annotations differ from Labels in terms of image metadata locality/terminology, but they do here.

  Annotations: 
    org.opencontainers.image.created:         2024-08-17T04:06:17Z

docker buildx imagetools inspect ros:noetic

$ docker buildx imagetools inspect ros:noetic
Name:      docker.io/library/ros:noetic
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:e3866df3b9798c4f3b49946d61b179745a22409ee8ce7037411ea550581b12b4
           
Manifests: 
  Name:        docker.io/library/ros:noetic@sha256:c7631b6323509f943142bd494abf268127a6807d81bd75e5f1fec2e0cd2caf7f
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
  Annotations: 
    org.opencontainers.image.created:         2024-08-17T04:06:17Z
    org.opencontainers.image.revision:        df19ab7d5993d3b78a908362cdcd1479a8e78b35
    org.opencontainers.image.source:          https://github.com/osrf/docker_images.git#df19ab7d5993d3b78a908362cdcd1479a8e78b35:ros/noetic/ubuntu/focal/ros-base
    org.opencontainers.image.url:             https://hub.docker.com/_/ros
    org.opencontainers.image.version:         noetic-ros-base
    com.docker.official-images.bashbrew.arch: amd64
    org.opencontainers.image.base.digest:     sha256:e53f9fd13c91747c3515c3a85ec56075ca7a514c7363dd29953d77ac708f8dde
    org.opencontainers.image.base.name:       ros:noetic-ros-core-focal
               
  Name:        docker.io/library/ros:noetic@sha256:fc58b4b5b47a9daf00a015010b9c405b2bee15fbcedb748dc03ec81728c088fc
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    com.docker.official-images.bashbrew.arch: amd64
    vnd.docker.reference.digest:              sha256:c7631b6323509f943142bd494abf268127a6807d81bd75e5f1fec2e0cd2caf7f
    vnd.docker.reference.type:                attestation-manifest
               
  Name:        docker.io/library/ros:noetic@sha256:16f26c7d44fbd3cf3d9e035ff76de360e992cf6fc7e4254290980286774ed4d3
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v7
  Annotations: 
    org.opencontainers.image.base.name:       ros:noetic-ros-core-focal
    org.opencontainers.image.created:         2024-08-17T04:36:03Z
    org.opencontainers.image.revision:        df19ab7d5993d3b78a908362cdcd1479a8e78b35
    org.opencontainers.image.source:          https://github.com/osrf/docker_images.git#df19ab7d5993d3b78a908362cdcd1479a8e78b35:ros/noetic/ubuntu/focal/ros-base
    org.opencontainers.image.url:             https://hub.docker.com/_/ros
    org.opencontainers.image.version:         noetic-ros-base
    com.docker.official-images.bashbrew.arch: arm32v7
    org.opencontainers.image.base.digest:     sha256:118b7e318e80c9dd3db19659dff04d5107aeffae11996479a82e754068453b0b
               
  Name:        docker.io/library/ros:noetic@sha256:2df235d55b1f2af746ffc7217df4e45535de5e5073e1f5827e6105ab6b528b73
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    com.docker.official-images.bashbrew.arch: arm32v7
    vnd.docker.reference.digest:              sha256:16f26c7d44fbd3cf3d9e035ff76de360e992cf6fc7e4254290980286774ed4d3
    vnd.docker.reference.type:                attestation-manifest
               
  Name:        docker.io/library/ros:noetic@sha256:52faca237a809534ad5a7ed7e690f03870c886d409b7f36a7f023fb2c78c8dd5
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations: 
    org.opencontainers.image.base.name:       ros:noetic-ros-core-focal
    org.opencontainers.image.created:         2024-08-17T08:07:06Z
    org.opencontainers.image.revision:        df19ab7d5993d3b78a908362cdcd1479a8e78b35
    org.opencontainers.image.source:          https://github.com/osrf/docker_images.git#df19ab7d5993d3b78a908362cdcd1479a8e78b35:ros/noetic/ubuntu/focal/ros-base
    org.opencontainers.image.url:             https://hub.docker.com/_/ros
    org.opencontainers.image.version:         noetic-ros-base
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:743c284f23157c9b9e31d8c0bc92b6d4904a4c7ddcbf41bd57a18d090b97e2f3
               
  Name:        docker.io/library/ros:noetic@sha256:a219de600f12d49f47f8089bc52b3165408aaa6070138f27f22a39ff303b087f
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    com.docker.official-images.bashbrew.arch: arm64v8
    vnd.docker.reference.digest:              sha256:52faca237a809534ad5a7ed7e690f03870c886d409b7f36a7f023fb2c78c8dd5
    vnd.docker.reference.type:                attestation-manifest

docker image inspect ros:noetic

$ docker image inspect ros:noetic
[
    {
        "Id": "sha256:d408a67701899f969b3e60f4a0eb58d35defafadadb7f564aadcd939bac6d28d",
        "RepoTags": [
            "ros:noetic"
        ],
        "RepoDigests": [
            "ros@sha256:e3866df3b9798c4f3b49946d61b179745a22409ee8ce7037411ea550581b12b4"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2020-11-17T19:36:01Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "LC_ALL=C.UTF-8",
                "ROS_DISTRO=noetic"
            ],
            "Cmd": [
                "bash"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/ros_entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "20.04"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 933267192,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/d48690391209f7e58ba0779715b853c84b886544924d1fef359771b4678804e3/diff:/var/lib/docker/overlay2/0ebb49014f9ba33d29896952d1b7d65741318fc7fc1a34b43bd9a2a30f815e81/diff:/var/lib/docker/overlay2/f17489bfe791fecca65484880cc46b07b1b405e4828b8cbd5c69a85f1bb5b0e3/diff:/var/lib/docker/overlay2/94aaa8509aaf8f5f45fe51b70b346995669096d51dd89f3a4373abf3db231488/diff:/var/lib/docker/overlay2/27d3a86f7a970999cff8cb4377f722e76a71be1a84b3fee0ed4562856ac91bf2/diff:/var/lib/docker/overlay2/691d5d5361f3034faddc965a14a34ae117434b44010335c3c65a65951f11ee0c/diff:/var/lib/docker/overlay2/4435f44e2a34057389aa3a82272a3690ccd3c730c6770bdd2470dcbee66484d0/diff:/var/lib/docker/overlay2/e9acf39376f84e8a6aafe7e9d032c87dacc7195bdef7fac797a1f1cd67589521/diff:/var/lib/docker/overlay2/afbe150258784e39214ab1de2d973f38ced7de8b3072784cebe6b77d1a51aa2f/diff",
                "MergedDir": "/var/lib/docker/overlay2/29c3ffbc71d0446119da46d1bf9a1c1c4db8962ea2059ccc4fd848b81c25ba79/merged",
                "UpperDir": "/var/lib/docker/overlay2/29c3ffbc71d0446119da46d1bf9a1c1c4db8962ea2059ccc4fd848b81c25ba79/diff",
                "WorkDir": "/var/lib/docker/overlay2/29c3ffbc71d0446119da46d1bf9a1c1c4db8962ea2059ccc4fd848b81c25ba79/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:3ec3ded77c0ce89e931f92aed086b2a2c774a6fbd51617853decc8afa4e1087a",
                "sha256:fc249c6b471458c0c9d69ef39580fd2bf3946f601a6217a8e54415a9b080a6eb",
                "sha256:7cacef60b7fc340ae9ea66ab84fc0747805f09cb816cc5b5283178584b6b2fdb",
                "sha256:5de4e0048b24121099af45f5c26b54ed850dba56d26efa8ac273e49f07030231",
                "sha256:a68a9aad0b492719fe90d6740237b7c75533e858b2b90040ae32c7dd531135fb",
                "sha256:96759a6874dbe99356b1a345a21e7dfc0161bd4e000e7e051d44e621c9813cd0",
                "sha256:2f9d32a36314d41ef3bc31f2f82242cf57b131e113f15b1d737ab860290a8945",
                "sha256:7a7a9714dc3e6fc7789fa6875c81779a59218c2da23caaf3d1f835026e53755f",
                "sha256:38b803f6d4ddf290f02a01c391b62025da713d4a36649202ac7f3e22017d16a0",
                "sha256:0f9e192c384c890876739a431490cb1428d4dd07d777d5904672708b82d3bf1b"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

I briefly went down this rabbit hole back when support for mult arch manifest lists was new, and more recently while trying to fetch layer shas from the registry without pulling the image (but was only able to query the compressed shas).