Windows docker login, "stub received bad data"

[djeikyb]

Description

On windows server 2022, when using docker login so I can push an image to ecr, I get this weird error:

Error saving credentials: error storing credentials - err: exit status 1, out: `The stub received bad data.`

There's lots of recommendations to clear or remove the .docker\config.json file, but that doesn't fix the issue.

Reproduce

1. Start a windows 2022 server ec2 instance in aws.
2. Install docker for windows
3. Switch to windows containers
4. (Get-ECRLoginCommand).Password | docker login --username AWS --password-stdin my-ecr-am.amazonaws.example.com

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:53:15 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.20.0 (109717)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.24)
  Go version:       go1.20.4
  Git commit:       659604f9
  Built:            Thu May 25 21:52:13 2023
  OS/Arch:          windows/amd64
  Experimental:     false

docker info

Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 8
  Running: 0
  Paused: 0
  Stopped: 8
 Images: 74
 Server Version: 24.0.2
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 20348 (20348.1.amd64fre.fe_release.210507-1500)
 Operating System: Microsoft Windows Server Version 21H2 (OS Build 20348.1547)
 OSType: windows
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.86GiB
 Name: EC2AMAZ-H68PGQ1
 ID: 3cc58b8c-583d-4ff9-9632-529b7596c882
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[djeikyb]

I finally found someone recommending to run the system file checker. It worked!

sfc /scannow

Sorry for the spam, I just spent far too long on this. Hopefully this helps someone out?

[thaJeztah]

Interesting; hadn't heard of that one (but admitted, I'm not on Windows myself). IIUC that command is to fix issues with your Windows installation (as a whole?)

[djeikyb]

Myself, I'm on a mac 😄 Just over here tryna deploy legacy asp.net applications lol.

And yeah, I think there must be something weird with the win server 2022 image from aws. The problem was only manifesting when trying to use docker login, but that sfc fixed things up, i'm pretty sure means it wasn't related to docker.

[thaJeztah]

Thanks! I'll try to keep this one in the back of my mind in case I see someone reporting something like this!

[crazy-max]

Hum also encounter this one on windows runners on GHA: https://github.com/docker/login-action/actions/runs/5472897107/jobs/9965664777#step:4:15

[thaJeztah]

Let me post that output here as well (easier to search);

Run ./
  with:
    registry: public.ecr.aws
    ecr: auto
    logout: true
  env:
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
Retrieving registries data through AWS SDK...
AWS Public ECR detected with us-east-1 region
Logging into public.ecr.aws...
Error: Error saving credentials: error storing credentials - err: exit status 1, out: `The stub received bad data.`

[thaJeztah]

Some mention in Go code here; https://github.com/minepkg/minepkg/blob/c2146e9b476e6e2ed46a23f21b89a9aaff132c25/internals/auth/microsoft.go#L23-L32

// MicrosoftCredentialStorage is used to trim down the auth data to the minimum required
// otherwise the windows keyring will return an error ("The stub received bad data.")
type MicrosoftCredentialStorage struct {
	MicrosoftAuth oauth2.Token `json:"ms"`
	PlayerName    string       `json:"pn"`
	UUID          string       `json:"id"`
	AccessToken   string       `json:"at"`
	ExpiresAt     time.Time    `json:"exp"`
	XUID          string       `json:"xuid,omitempty"`
}

Error-code (not the canonical source of truth); https://github.com/mickael-kerjean/filestash/blob/c2059c839d516c2ee56570d1d636c97a9c807278/vendor/github.com/hirochachacha/go-smb2/internal/erref/ntstatus.go#L2938

RPC_NT_SS_CANNOT_GET_CALL_HANDLE:                                   "The stub is unable to get the call handle.",

But looks to be documented here; https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55

[yerlibilgin]

Hi people. @djeikyb I am sorry to see that you have closed the report. After all, sfc /scannow is not a general solution, but a workaround. In recent windows versions, this has been a quite problem where people devise custom workarounds for it (deploying alternative credential helpers, which, I believe, isn't a quite sustainable solution). So I would love to see it reopened and solved.

I believe the issue is that the docker windows credential manager doesn't like looong stuff. Maybe they forgot to include tokens, and focused only in human passwords of some maximum length.

[djeikyb]

@yerlibilgin tbh i opened this issue after i already had a solution just to document (a) the problem (b) the solution i painstakingly discovered. if you think this is a problem that should be solved in the docker codebase, please do open an issue! and tag this one in for searchability!

[criscola]

Unfortunately sfc /scannow did not fix the issue for me.

[thaJeztah]

I'm re-reading the issue, and the comment I left further up #346, and I think I have a slightly clearer picture of the issue (bear with me; I'm not a Windows user), although I don't exactly understand how the sfc /scannow relates.

The original issue was reported with this command;

(Get-ECRLoginCommand).Password | docker login --username AWS --password-stdin my-ecr-am.amazonaws.example.com

This command;

• calls (Get-ECRLoginCommand).Password, which produces a token / password
• which gets piped to docker login, passing the token / password to it through stdin
• and calls the registry (my-ecr-am.amazonaws.example.com) to verify the credentials
• on success, it stores the credentials
  • without a credentials-helper, those credentials are stored in plain text (base64-encoded) in the CLI config file (~/.docker/config.json)
  • with a credentials-helper those credentials are sent to the credentials-helper which, on windows, is the wincred credentials-helper (https://github.com/docker/docker-credential-helpers/tree/master/wincred) which stores it in the Windows Credentials Store.

From my understanding, the last part is where things fail, and from #346 it looks that (as also mentioned by @yerlibilgin #346) the The stub received bad data. error is produced by the Windows API to store those credentials if the data is too large.

I'm not sure if there's good ways around this; the code in the minepkg/minepkg project (from trying to read their changes) worked around this issue by stripping the token from data that can be omitted, but in their case it's a token they produce and therefore were able to mutate.

In the docker CLI + credentials helper case, that's not a real option; the CLI treats the password (or token passed as password) as opaque data; it verified that data is correct against the registry in which case it stores it as-is. And, in this case, the token is produced by (Get-ECRLoginCommand).Password, which is a 3rd party tool, so we cannot mutate that data.

So the only solution I could see here is to either

• fix (Get-ECRLoginCommand).Password to produce a token that's smaller (I guess it may depend on the situation what information is captured in the token)
• find if the Windows Credentials Store allows different ways to store data that's exceeding the size (we should check if the size limit is documented so that we could produce a more useful error).

That last change would have to be implemented in either the credentials helper (https://github.com/docker/docker-credential-helpers/tree/master/wincred), or the Go library that's used to interact with the Windows Credentials store (github.com/danieljoos/wincred).

I will reopen this ticket, but move it to the https://github.com/docker/docker-credential-helpers repository's issue tracker, because the underlying issue here does not appear to be in the docker CLI itself.

Let me also cc @austinvazquez in case he has more insight in the ECRLoginCommand to see if there's options for it to produce a token that's smaller (omitting data that may not be essential?)

[fartosh]

@thaJeztah thanks for sharing, it helped me find the solution - in my case I simply removed "credsstore": "wincred" from %APPDATA%\helm\registry\config.json because I didn't need it.

From my understanding, the last part is where things fail, and from #346 it looks that (as also mentioned by @yerlibilgin #346) the The stub received bad data. error is produced by the Windows API to store those credentials if the data is too large.

I can confirm that. Today I faced the same issue with helm registry login public.ecr.aws. I checked that in my %APPDATA%\helm\registry\config.json there was Windows Credential Manager configured as a store.

we should check if the size limit is documented so that we could produce a more useful error

Not really an official documentation but if you go to Credential Manager and try to add a generic credential, the password field will remain blank if you try to paste the generated token but it will accept anything that is significantly shorter. On my machine (Windows 10 22H2) the limit is set to 259 characters, the password field will not accept any string equal to or longer than 260 characters.

[davecranwell]

I got this for the first time today. I was running WSL2 and trying to build for ECR from there, when I experienced the problem.

All the solutions I found suggested removing the credsStore value of your docker config.json. But it wasn't the Windows version of that file (at %UserProfile%\.docker\config.json) I had to change but rather the Linux one, on WSL2 at `~/.docker/config.json.

In my case I found that credsStore in the linux file had the value desktop.exe and removing this line entirely solved my problems! Changing the windows version just resulted in the same data being added back to it every time docker desktop restarted.