correct the docker pushes (#1063)

So, since #1060, we're not able to do deploys. They error like as in:
https://github.com/mozilla-services/autograph/actions/runs/11748424011/job/32732461427

I thought the `sha` I was removing was the hash of the image, but it's
the git commit's sha!

The real problem seems to be we are pushing to both dockerhub and to
Google Artifact Registry (GAR) at the same time after our builds on
`main`. We wanted only to push to GAR and then on release copy them to
dockerhub.

That happens becuase the `docker/metadata-action` action gets called
with both Docker Hub and GAR in its `images` argument.

So, to fix this, I thought really hard about what we want to happen.

We currently have two separate deploy systems. In one, we need to push
non-`latest` tags to Docker Hub only when we want a staging deploy to
occur. In the other, we need to push to a semver-like tag to Google
Artifact Registry only when we want a staging release to occur.
Production deploys in both require a human to intervene and change the
expected deployed version.

In the deploy system that uses GAR, we need a push to `latest` on each
commit to effect a dev deploy.

We also use the `latest` on Docker Hub to represent the current `main`
branch's code for use in other people's projects.

So, let's sum that up in terms of tags.

On each commit to `main`, we want these docker tags and only
these docker tags pushed up:

* `:latest` to `GAR`
* `:sha-<the git commit's sha>` to GAR
* `:latest` to Docker Hub

On each release, we want these docker tags and only these docker tags
pushed up:

* `:<git tag of release>` to GAR
* `:<git tag of release>` to Docker Hub

Those `:<git tag of release>` will be copies of the `sha-<the git
commit's sha>` tag that was made on commit to `main` and pushed to GAR.

This patch does that by breaking up single `docker/metadata-action`
action call to two. One for GAR and one for Docker Hub; both with
different sets of tags depending on whether this is a "push" (commit on
main) or "release" event.

But doing just that doesn't quite solve it. Becuse we use
`docker/build-push-action` to both build and push to GAR in one go. To
avoid building on release, we have to skip that step, but that step is
what was doing all the work of talking to GAR. So, we also need to, only
on release, "copy" the previously made `sha-` tag to the `:<git tag of
release>` tag within GAR itself.

This may be the point where we should split this one GitHub Actions
workflow into separate `deploy` and a `release` workflows. There is a
lot of thinking and conditional checks. It would cost a significant
amount of duplication between these two actions to do so. But I could be
told to do so.

(The crane install would be especially annoying; maybe we
should just go install it, anyway, since this install instructions were
for a non-Go environment)

.github/workflows/deploy.yaml

@@ -26,18 +26,31 @@ jobs:
       - name: Setup Docker Buildx
         uses: docker/setup-buildx-action@v3
 
-      - name: Docker Metadata
-        id: meta
+      - name: Docker Metadata for Google Artifact Registry build
+        id: metagar
         uses: docker/metadata-action@v5
         with:
           flavor:
             # don't automatically tag with `latest`; we do this conditionally in the `tags` section
             latest=false
           images: |
-            ${{ vars.DOCKERHUB_REPO }}
             ${{ vars.GAR_LOCATION }}-docker.pkg.dev/${{ vars.GCP_PROJECT_ID }}/${{ vars.GAR_REPOSITORY}}/autograph
           tags: |
-            type=semver,pattern={{raw}}
+            type=sha,format=long,enable=${{ github.event_name == 'push' }}
+            type=raw,value=latest,enable=${{ github.event_name == 'push' }}
+            type=semver,pattern={{raw}},enable=${{ github.event_name == 'release' }}
+
+      - name: Docker Metadata for Docker Hub
+        id: metahub
+        uses: docker/metadata-action@v5
+        with:
+          flavor:
+            # don't automatically tag with `latest`; we do this conditionally in the `tags` section
+            latest=false
+          images: |
+            ${{ vars.DOCKERHUB_REPO }}
+          tags: |
+            type=semver,pattern={{raw}},enable=${{ github.event_name == 'release' }}
             type=raw,value=latest,enable=${{ github.event_name == 'push' }}
 
       - name: Generate version.json
@@ -64,16 +77,16 @@ jobs:
           username: ${{ vars.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_PASSWORD }}
 
-      - name: Build and push
+      - name: Build and push to Google Artifact Registry for each commit on main
         # On pushes to `main`, we build and push a new image, so we can simply
         # use the `docker/build-push-action` action.
         if: ${{ github.event_name == 'push' }}
         uses: docker/build-push-action@v6
         with:
-          push: ${{ github.event_name == 'push' }}
+          push: true
           sbom: true
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
+          tags: ${{ steps.metagar.outputs.tags }}
+          labels: ${{ steps.metagar.outputs.labels }}
           context: .
           platforms: linux/amd64,linux/arm64
 
@@ -95,17 +108,42 @@ jobs:
           echo "${tmp}" >> $GITHUB_PATH
           echo "${{ github.token }}" | crane auth login ghcr.io --username "dummy" --password-stdin
 
-      - name: Tag and push
-        # For releases, we specifically do _not_ want to rebuild, just tag the
-        # existing image and push. There's no officially maintained action for
-        # this use case, but it's trivial enough to do ourselves.
+      - name: Copy from Google Artifact Registory to Docker Hub
+        # We want to make sure that Docker Hub's latest tag is always up to
+        # date. For releases, we specifically do _not_ want to rebuild and
+        # instead only tag the existing image and push. There's no officially
+        # maintained action for this use case, but it's trivial enough to do
+        # ourselves. On builds on each commit of main ("push" events), the
+        # expectation is that only `latest` will be pushed to Docker Hub. For
+        # releases, the expectation is only the semver tag will be pushed.
+        env:
+          # Tags come in the form of a fully qualified image name and tag
+          # separated by newlines.
+          TAGS: |
+            ${{ steps.metahub.outputs.tags }}
+          # This github.sha won't work if the tag is an annotated tag, but we'll
+          # hit that when we hit that. (We'll want to use github.object.sha but
+          # only in tags, not in pushes)
+          SRC: ${{ vars.GAR_LOCATION }}-docker.pkg.dev/${{ vars.GCP_PROJECT_ID }}/${{ vars.GAR_REPOSITORY}}/autograph:sha-${{ github.sha }}
+        run: |
+          crane digest $SRC
+          crane manifest $SRC
+          for tag in $TAGS; do
+            crane copy $SRC $tag
+          done
+
+      - name: Copy image from Google Artifact Registory to Google Artifact Registory as semver tag
+        # This is conditional because we only can do this correctly (that is,
+        # with the correct TAGS from the metagar step) and only need to do it at
+        # all on release.
         if: ${{ github.event_name == 'release' }}
         env:
-          # Tags come in the form of a fully qualified image name and tag, eg:
-          # mozilla/autograph:1.1.8
-          # us-west2-docker.pkg.dev/autograph-proj/autograph-repo/autograph:1.1.8
-          TAGS: ${{ steps.meta.outputs.tags }}
-          SRC: ${{ vars.DOCKERHUB_REPO}}:sha-${{ github.sha }}
+          TAGS: |
+            ${{ steps.metagar.outputs.tags }}
+          # This github.sha won't work if the tag is an annotated tag, but we'll
+          # hit that when we hit that. (We'll want to use github.object.sha but
+          # only in tags, not in pushes)
+          SRC: ${{ vars.GAR_LOCATION }}-docker.pkg.dev/${{ vars.GCP_PROJECT_ID }}/${{ vars.GAR_REPOSITORY}}/autograph:sha-${{ github.sha }}
         run: |
           crane digest $SRC
           crane manifest $SRC