Working with Github package registry

[kyler-hyuna]

Hey so i'm currently stuck between v1 no longer working and v2 not working either.

I use the Github container registry in a private repo.

In v1 i used:

- uses: docker/build-push-action@v1
  with:
    username: ${{ github.actor }}
    password: ${{ github.token }}
    registry: docker.pkg.github.com
    repository: <org-name>/<repo-name>/<app-name>
    tag_with_ref: true

This worked great. However this is failing recently. I suspect a major update to docker happened cause I cannot even pull these images locally since it fails with the error:

Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

In v2:
I'm using the setup from the readme:

- run: echo ::set-env name=IMAGE_TAG::${GITHUB_REF:10}

- name: Set up QEMU
  uses: docker/setup-qemu-action@v1
  with:
    platforms: all

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v1

- name: Cache Docker layers
  uses: actions/cache@v2
  with:
    path: /tmp/.buildx-cache
    key: ${{ runner.os }}-buildx-${{ github.sha }}
    restore-keys: |
      ${{ runner.os }}-buildx-

- name: Login to Registry
  uses: docker/login-action@v1
  with:
    registry: docker.pkg.github.com
    username: ${{ github.actor }}
    password: ${{ github.token }}

- uses: docker/build-push-action@v2
  with:
    push: true
    tags: <org-name>/<repo-name>/<app-name>:${{ env.IMAGE_TAG }}
    secrets: |
      GIT_AUTH_TOKEN=${{ github.token }}

This fails with:

failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed

Is there anything i am missing here?

[crazy-max]

@kyler-hyuna Do you have a link to your repo? Also docker.pkg.github.com is deprecated and will sunset early next year. I suggest to migrate to GitHub Container Registry instead.

[hiddeco]

We had this happen for 4 images in total for ghcr.io, all build and released on different dates. Retriggering the release action (which overwrote the corrupt tag) resolved the issue.

$ docker pull ghcr.io/fluxcd/source-controller:v0.0.16
v0.0.16: Pulling from fluxcd/source-controller
df20fa9351a1: Already exists 
fbd8fc656b09: Pulling fs layer 
fc19d3ac910e: Pulling fs layer 
61af7bc6ae34: Pulling fs layer 
81d1fa28d260: Waiting 
error pulling image configuration: unknown blob

Failed to pull image "ghcr.io/fluxcd/source-controller:v0.0.16": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/fluxcd/source-controller:v0.0.16": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:7f5a4353bb77648f85a1e79000522bcfc0e0ae1bb75f9f163ee4d5715a408802 (application/vnd.docker.container.image.v1+json) from remote: not found

Affected packages:

https://github.com/orgs/fluxcd/packages/container/package/helm-controller
https://github.com/orgs/fluxcd/packages/container/package/kustomize-controller
https://github.com/orgs/fluxcd/packages/container/package/notification-controller
https://github.com/orgs/fluxcd/packages/container/package/source-controller

Builds that produced the corrupt images:

https://github.com/fluxcd/helm-controller/actions/runs/249887657
https://github.com/fluxcd/kustomize-controller/actions/runs/253846609
https://github.com/fluxcd/source-controller/actions/runs/251027151
https://github.com/fluxcd/notification-controller/actions/runs/251041063

Apparently GitHub action re-runs overwrite all data from the previous run 🤦

[kyler-hyuna]

@crazy-max I tried github container registry as well but it failed. Build-x returns

failed to solve: rpc error: code = Unknown desc = unexpected response: 401 Unauthorized

But v2 worked perfectly with aws ecr, so i suspect it's some mismatch between github's repositories and buildx

[crazy-max]

@kyler-hyuna @hiddeco Someone from GitHub told me they had an issue with the new cross repo mounting and should have been fixed about an hour ago.

[crazy-max]

@kyler-hyuna Do you have a link to your repo? Thanks.

[ghostwriter]

@kyler-hyuna your issue looks like an authentication 401 Unauthorized and maybe tags.

If you're planning on migrating to GitHub Container Registry,

1. Create a new personal access token (PAT) with the appropriate scopes for the tasks you want to accomplish. If your organization requires SSO, you must enable SSO for your new token.

• Select the read:packages scope to download container images and read their metadata.
• Select the write:packages scope to download and upload container images and read and write their metadata.
• Select the delete:packages scope to delete container images.

2. In your GitHub Actions workflow files, update the package URL from https://docker.pkg.github.com to ghcr.io.

3. Add your new container registry authentication personal access token (PAT) as a GitHub Actions secret. GitHub Container Registry does not support using GITHUB_TOKEN for your PAT so you must use a different custom variable, such as CR_PAT. For more information, see "Creating and storing encrypted secrets."

---

Updated build.yml

- run: echo ::set-env name=IMAGE_TAG::${GITHUB_REF:10}

- name: Set up QEMU
  uses: docker/setup-qemu-action@v1
  with:
    platforms: all

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v1

- name: Login to Registry
  uses: docker/login-action@v1
  with:
    # registry: docker.pkg.github.com
    registry: ghcr.io
    username: ${{ github.repository_owner }}
    password: ${{ secrets.CR_PAT }}

- name: Cache Docker layers
  uses: actions/cache@v2
  with:
    path: /tmp/.buildx-cache
    key: ${{ runner.os }}-buildx-${{ github.sha }}
    restore-keys: ${{ runner.os }}-buildx-

- uses: docker/build-push-action@v2
  with:
    push: true
    # tags: <org-name>/<repo-name>/<app-name>:${{ env.IMAGE_TAG }}
    tags: ghcr.io/${{ github.repository_owner }}/<image-name>:${{ env.IMAGE_TAG }}

I hope you find this helpful. ✌🏽 @crazy-max @hiddeco @kyler-hyuna

[kyler-hyuna]

@crazy-max private company repo sorry

@nathane Followed these exact steps but without success. Is anything different when organizations are used? It just says

unauthorized
##[error]Process completed with exit code 1.

[crazy-max]

@kyler-hyuna It seems you cannot use GitHub Container Registry in private repos:

GitHub Packages is not available for private repositories owned by accounts using legacy per-repository plans. Also, accounts using legacy per-repository plans cannot access GitHub Container Registry since these accounts are billed by repository. For more information, see "GitHub's products."

See https://docs.github.com/en/packages/managing-container-images-with-github-container-registry/configuring-access-control-and-visibility-for-container-images

I suggest you open a support ticket to GitHub about this: https://support.github.com/contact

[kyler-hyuna]

Thanks so much man, been pulling hair on this.

[clarkbw]

Thanks so much man, been pulling hair on this.

Can you reach out to support so we can dig into this? Neither the old docker service, nor GHCR work with GitHub Legacy accounts. However with recent changes many accounts can be upgraded to Teams or other account levels for very little change in billing; depending on circumstances.

[LaysDragon]

@kyler-hyuna Do you have a link to your repo? Also docker.pkg.github.com is deprecated and will sunset early next year. I suggest to migrate to GitHub Container Registry instead.

That really confused me while I known the ghcr in the first place,since it almost the same thing with Package Service
And now I known docker.pkg.github.com is going to deprecated, so we have to migrated to GitHub Container Registry in the feature.
But that so wired, it looks like migrated to GHCR also kill the integration of original github package design with repo ,and make things complex again. Because it doesn't support GITHUB_TOKEN anymore 🤔

[clarkbw]

Because it doesn't support GITHUB_TOKEN anymore

This is only during the Beta, we'll roll out support for this next.

[ianfixes]

Are the QEMU and Buildx steps required whenever docker/build-push-action@v2 is used? This isn't made plain in any of the docs I've read. Is there no longer a concept of a single action that takes code + creds and docker-pushes to a default github-owned docker repo (one that aligns with the github org and repo running the action)?

[crazy-max]

@ianfixes

Are the QEMU and Buildx steps required whenever docker/build-push-action@v2 is used? This isn't made plain in any of the docs I've read.

setup-buildx action is recommended to be able to push through docker-container driver. See #100 (comment).

setup-qemu is recommended if you want to add support for more platforms.

Is there no longer a concept of a single action that takes code + creds and docker-pushes to a default github-owned docker repo

See #208 (comment)

[jamesdh]

GitHub Container Registry absolutely works w/ private organizational repos. I've been able to push to my container registry via multiple methods. The comment @crazy-max refers to above, e.g.

GitHub Packages is not available for private repositories owned by accounts using legacy per-repository plans

is for legacy plans. I don't think you'll even see packages as an option if you're still on one of those plans.

@kyler-hyuna I fought with this a bit today when trying to switch to this action to better take advantage of layer/stage caching. You really need to make sure your tag correct, e.g. ghcr.io/orgname/imagename. But I can confirm, this action works just fine w/ GitHub Container Registry on a private repo within an organization..