Using 'docker push' to overwrite a tag fails to collect correct build-infos

[s-geiger-si]

Describe the bug

Hello JFrog Team,

we just noticed an interesting change in behavior of the JFrog CLI that appeared with the upgrade of the CLI from major version 1.x to 2.x. For us this causes a regression when we try to promote a docker image tag that already existed and that was overwritten in a recent push. This now leads to a warning and no build-info metadata being attached to the image, which causes the image layers to be not correctly promoted.

Our setup is as following, we have three repositories:

1. A virtual repo (i.e., docker).
2. Two local repos (i.e, docker-dev, docker-stable).
3. The virtual repo references the local repos in the order stable, then dev. Thus it stable images will shadow dev images, this could be part of the issue we are seeing.

We push an image with a tag such as foobar:latest which will first appear in docker-dev and then we want to promote it to docker-stable. Then later we run another pipeline, push, and try to promote to stable again, overwriting the older tag. But after the pipeline completes, we are still seeing the original image tag in the stable repository, while the new tag remains in the dev repository.

This used to work fine with the CLI version 1.x, However, the docker push command accepted a repository name in the 1.x branch, but as far as I can see it no longer allows this argument in the 2.x branch. So we removed this argument and now the following happens:

1. When we call jf docker push ... the image is pushed and a warning is emitted:

   12:41:11 [Warn] Failed to collect build-info, couldn't find image ".../foobar:latest" in Artifactory

   This warning seems to be caused by the following code: d45734b#diff-e2f77381b07f59446c81eef9ee262f955593314bd6328e0ed5fa4f8e350c0079R55

2. The docker image is correctly pushed, but the build-info contains no modules. When we then try to promote the build to the stable stage, the image layers remain in the dev stage.

I am not sure how we should fix this, or if this is even intentional behavior of the JFrog CLI. The problem, was far as I understand seems to come from the fact, that the repository is now determined using a call to the REST API and this API seems to return the docker-stable repository, since that one appears first in the order of the virtual repository and already contains an older image tag but the hashsum of that image does not match the one we just pushed. The result is that we are not attaching any build-info to the newly pushed image tag.

Current behavior

As described above, the build-info metadata is not correctly attached to the image tag and it is not correctly promoted to the stable docker repository stage as a result.

Reproduction steps

No response

Expected behavior

The image correctly receives the build-info metadata.

JFrog CLI-Core version

2.28.0

JFrog CLI version (if applicable)

2.28.1

Operating system type and version

Linux

JFrog Artifactory version

7.49.10

JFrog Xray version

No response

[s-geiger-si]

Also here is another link to the code, where the repository is being determined using the REST API:
https://github.com/jfrog/jfrog-cli-core/blob/d45734b8f3723c55ff9cefb6c7eb151aee7c7ef8/artifactory/utils/container/image.go#LL143C60-L143C60

[Or-Geva]

Hello @s-geiger-si, and thank you for taking the time to reach out to us regarding JFrog CLI and Docker.

First and foremost, I want to emphasize the utmost importance of maintaining accurate build info. Build info serves as a reliable snapshot of the components used in the build process, enabling traceability, analysis, and ensuring the security of your software. It is critical that the build info remains unchanged and accurately reflects the build metadata. If JFrog CLI encounters any discrepancies, such as an incorrect checksum, it will not generate build-info data.

Upon reviewing your case, it seems that you are attempting to overwrite a Docker image with the same name and tag in Artifactory. However, during this process, JFrog CLI encounters different layer SHA values that were not included in the initial push operation to Artifactory. As a result, the build-info data cannot be generated due to these discrepancies.

The removal of the repository name argument in the Docker push command in JFrog CLI from version 1.x to 2.x was driven by two primary motivations:

Convenience: Our goal is to simplify and streamline the CLI commands for our users. By reducing the number of required arguments, we aim to provide a more convenient interaction with JFrog CLI.

Enhanced Security: The removal of the repository name argument contributes to a more secure build-info generation process. In the previous version, specifying the repository name during Docker push could potentially allow manipulation or misconfiguration, leading to inaccurate or compromised build info. By relying on Artifactory REST API to determine the repository based on the image tag, we ensure a more robust and secure approach to build-info generation.

Please accept my apologies for any inconvenience caused. If you have any further questions or concerns, please don't hesitate to reach out.

[s-geiger-si]

Hello @Or-Geva,

thanks for the detailed response. I absolutely agree, that accurate build-info is very important and we try to create our pipelines in such a way that they include all relevant details and submit them together with the relevant artifacts.

I understand the motivation that lead to the removal of the repository argument, I am not asking to bring this option back. However I am not sure how we should solve this issue that we have or if you have any alternative suggestion for us.

Let me try to explain a bit more details about this. Our repository is structured in such a way, that the virtual repository includes the docker-stable and also the docker-dev local repositories. The order of these is in such a way that the docker-stable is seen first followed by docker-dev. By default, we only interact with the virtual repository as the local repositories do not have an exposed port. So we cannot directly upload to local repositories. The virtual repository has a default target that points to docker-dev, so when we push an image to the virtual repository, it will appear in the docker-dev local repository. In order to get images into the docker-stable local repository, we need to promote a build to stable. We do that using the move semantics (as opposed to copying) during the build promote step.

For every image we usually push a unique version, such as (1.0.0, or 1.0.0-xyz) where xzy is some additional suffix that can indicate a release candidate or some other alpha or beta pre-release (according to semver.org). Additionally, we maintain a set of relative tags such as latest, which we update for certain images. For example, if I had pushed 1.0.0 last month and 1.1.0 today, then I want to update the latest tag to point to 1.1.0 rather than 1.0.0, we use delete/overwrite permission semantics to achieve this.

As I mentioned earlier, we cannot directly update images in the stable repo to overwrite the latest tag, and need to push the image via the virtual repo which places them into the docker-dev local repo first, then we need to promote it from there. So for a brief period of time the docker-stable local repository will have a tag that points to 1.0.0 while the docker-dev local repo has also a tag latest that points to 1.1.0.

Now my expectation would be the following, please let me know if I make incorrect assumptions here of if this is not a good way to handle this:

1. The virtual repo will return the image for 1.0.0 if latest is requested, since in the virtual repo the docker-stable local repository is sorted before the docker-dev local repo.
2. The new build-info for the updated latest tag points to layers in docker-dev.
3. The older build-info for the original latest tag still points to layers in docker-stable.
4. When I promote the build for the latest image from docker-dev to docker-stable the latest tag in docker-stable should be overwritten, image layers that are no longer reachable by the new tag should be removed from the updated latest tag. The original build-info for the older latest tag, may now show missing layers (does it?), but the new build-info for the updated latest tag should reference the correct images in docker-stable.

I would like to add that a possible work around to the issue is to use the build-docker-create option from the JFrog CLI, which still supports to pass the repository argument, this allows me to collect and publish the build-info. However, after the build-info is published, I still see errors for some modules and layers. So this is not a fully functional solution at the moment.

[s-geiger-si]

@Or-Geva Did you see my reply?

[Or-Geva]

Hello, @s-geiger-si. My apologies for the delayed response. I understand that there are several missing configuration details that prevent me from fully grasping the complete picture. One specific point you mentioned is that:

By default, we only interact with the virtual repository as the local repositories do not have an exposed port.

Without going too deep, given this information, one approach I would suggest is to provide access to the necessary resources, such as docker-stable and docker-dev, to the relevant users or teams, such as developers. This way, they can directly push and promote images in a more deterministic manner. On the consumer side, you can share the virtual repository, allowing them to easily fetch the latest images without needing to determine the correct repository to pull from. This approach aims to simplify the process for different individuals by granting access only to the repositories they need, while abstracting the complexity from others.

In general, overriding images seems wrong to me, as it may lead to unexpected results and it may be worth reconsidering this practice. Again, I may have not the full picture from my point of view, so I recommend reaching out to JFrog support. They will be able to thoroughly analyze your repository structure and configuration and provide personalized assistance in designing the most suitable approach for your specific requirements.

[s-geiger-si]

Thanks @Or-Geva,

I always thought it was the default approach of Artifactory to interact with the virtual repo and use the "default deployment repository" to target the dev repo and then promote from there. I will check if that is an option to interact directly with the local repos.

Regarding your comment on overwriting tags, we never overwrite concrete versions (i.e., foo:1.0.0, only relative tags such as latest. If I look at platforms such as Docker Hub, they have a similar approach, where you can access the same images using multiple tags and some of the change over time. The unique version is always fixed, but they have aliases to track images by the major and minor versions, which update (for example) when a new patch level update is released.

If you look at postgres image for example it can be accessed using the 14 tag which as of today is an alias for 14.8-alpine3.18, but in a few month it might point to 14.9-alpine3.19, same for latest.

14.8, 14, 14.8-bookworm, 14-bookworm (see https://hub.docker.com/_/postgres)

I don't think there currently is an alternative to overwriting the image tags on the Artifactory platform, to realize this use case, is there?

Depending on the use case, we need to either pull an image by its concrete version, or by a relative tag such as latest and when we use latest, then we expect that it points to the most recent image that was pushed.

[Or-Geva]

@s-geiger-si, have you considered using a different term instead of "latest" for the build and push operation, such as "dev-latest"? By doing so, you can reserve the traditional "latest" keyword for pull operations. This approach would ensure that JFrog CLI correctly names the pushed images without any overlap with the "docker-stable" latest tag.
WDYT?

[s-geiger-si]

@Or-Geva, I am not sure I fully understood your comment. I think we are already doing this (limited by the bug that I am describing). We have (among others) dev-latest, and latest as tags that we are updating (and promoting).

My assumptions are as following:

• docker-dev and docker-stable refer to (local) repositories, these are fixed and we only have those two (local) repositories besides the virtual one.
• latest, dev-latest, etc. refer to docker image tags, the actual image layers that these tags point to need to change over time.
• The dev-latest tag can remain in the docker-dev repo, but the latest tag needs to change in the docker-stable repo once a new image that is first pushed to docker-dev is promoted to docker-stable.

By doing so, you can reserve the traditional "latest" keyword for pull operations.

Not sure what you mean by this, I would still need to update the latest tag to point to the new image layers, no?