Publish to self hosted pub server - 401 on upstream version check

[leoshusar]

• pub version or flutter pub version: Pub 2.16.0-80.1.beta
• OS version: Windows 11
• Are you using the Chinese community mirror or a corporate firewall: no

I am using unpub server, but when I want to publish my package to it using the dart pub publish, it fails.

I add my token using dart pub token add https://pub.mydomain.xy/ <token> and verify everything is ok.

pub publish first tries to check published versions by calling GET https://pub.mydomain.xy/api/packages/test_package.
But when it doesn't exist on my server, it returns 302 with the upstream package url (as seen here).

cURL response:

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 302
< content-type: text/plain; charset=utf-8
< date: Tue, 11 Jan 2022 23:04:11 GMT
< location: https://pub.dev/api/packages/test_package
< server: dart:io with Shelf
< strict-transport-security: max-age=315360000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-xss-protection: 1; mode=block
< content-length: 0

Now the pub client tries to check the upstream URL, gets 401 and the token get's removed (I believe by this).

I tried to check GET https://pub.dev/api/packages/oauth2 manually with my token and got a HTML response with this, is this even right?

Error <code>MissingAuthentication</code> <a href="#error-missingauthentication" class="hash-link">#</a></h1>
<p>Authentication is required, please add <code>authorization</code> header.

Without any Authorization header I get 200, but with the Authorization header it returns that the header is not there at all.

When I re-add that token right before the actual package upload, it proceeds fine.

Logs:

PS C:\Projects\Dart\test_package> dart pub publish --verbose
FINE: Pub 2.16.0-80.1.beta
IO  : Spawning "cmd /c git --version" in C:\Projects\Dart\test_package\.
IO  : Finished git. Exit code 0.
    | stdout:
    | | git version 2.24.0.windows.2
    | Nothing output on stderr.
FINE: Determined git command git.
IO  : Spawning "cmd /c git rev-parse --show-toplevel" in C:\Projects\Dart\test_package\.
IO  : Finished git. Exit code 128.
    | Nothing output on stdout.
    | stderr:
    | | fatal: not a git repository (or any of the parent directories): .git
FINE: Archiving and publishing Instance of 'Package'.
MSG : Publishing test_package1.0.0 to https://pub.mydomain.xy:
    | |-- CHANGELOG.md
    | |-- LICENSE
    | |-- README.md
    | |-- analysis_options.yaml
    | |-- example
    | |   '-- test_package_example.dart
    | |-- lib
    | |   |-- test_package.dart
    | |   '-- src
    | |       '-- test_package_base.dart
    | |-- pubspec.yaml
    | |-- test
    | |   '-- test_package_test.dart
    | '-- test.rest
FINE: Creating .tar.gz stream containing:
    | .\test.rest
    | .\test\test_package_test.dart
    | .\README.md
    | .\pubspec.yaml
    | .\LICENSE
    | .\lib\src\test_package_base.dart
    | .\lib\test_package.dart
    | .\example\test_package_example.dart
    | .\CHANGELOG.md
    | .\analysis_options.yaml
... truncated
SLVR: fact: test_package importer is 0.0.0
SLVR: derived: test_package importer
IO  : Get versions from https://pub.mydomain.xy/api/packages/test_package.
IO  : HTTP GET https://pub.mydomain.xy/api/packages/test_package
    | Accept: application/vnd.pub.v2+json
    | authorization: <censored>
    | user-agent: Dart pub 2.16.0-80.1.beta
SLVR: fact: test_package importer depends on test_package 1.0.0 from path
SLVR:   selecting test_package importer
SLVR:   derived: test_package 1.0.0 from path
SLVR:     selecting test_package 1.0.0 from path
SLVR: Version solving took 0:00:00.132464 seconds.
    | Tried 1 solutions.
IO  : HTTP response 401 Unauthorized for GET https://pub.mydomain.xy/api/packages/test_package
    | took 0:00:01.608713
    | date: Tue, 11 Jan 2022 22:16:04 GMT
    | content-length: 8190
    | referrer-policy: no-referrer-when-downgrade
    | via: 1.1 google
    | content-type: text/html; charset="utf-8"
    | x-frame-options: SAMEORIGIN
    | x-xss-protection: 1; mode=block
    | x-content-type-options: nosniff
    | server: dart:io with Shelf

here I see in my server logs 302, so it must've tried to check the official pub.dev

IO  : Writing 25 characters to text file C:\Users\husar\AppData\Roaming\dart\pub-tokens.json.
FINE: Contents:
    | {"version":1,"hosted":[]}
WARN: Invalid token for https://pub.mydomain.xy deleted.

and my token is gone

WARN: Package validation found the following potential issue:
WARN: * It's strongly recommended to include a "homepage" or "repository" field in your pubspec.yaml
MSG :
    | Publishing is forever; packages cannot be unpublished.
    | Policy details are available at https://pub.dev/policy
FINE: Showing confirm message:
    |
    | Package has 1 warning.. Do you want to publish test_package 1.0.0


Package has 1 warning.. Do you want to publish test_package 1.0.0 (y/N)? y

Now since pub-tokens.json is empty now, it would fail.
But since it's waiting for my input, I can re-run dart pub token add https://pub.mydomain.xy/ and the rest of the process continues successfully.

MSG : Uploading...
IO  : HTTP GET https://pub.mydomain.xy/api/packages/versions/new
    | Accept: application/vnd.pub.v2+json
    | authorization: <censored>
    | user-agent: Dart pub 2.16.0-80.1.beta
IO  : HTTP response 200 OK for GET https://pub.mydomain.xy/api/packages/versions/new
    | took 0:00:00.059056
    | access-control-allow-origin: *
    | date: Tue, 11 Jan 2022 22:16:19 GMT
    | content-length: 76
    | strict-transport-security: max-age=315360000; includeSubDomains; preload
    | x-frame-options: DENY
    | content-type: application/json
    | x-xss-protection: 1; mode=block
    | x-content-type-options: nosniff
    | server: dart:io with Shelf
IO  : HTTP POST https://pub.mydomain.xy/api/packages/versions/newUpload
    | authorization: <censored>
    | content-type: multipart/form-data; boundary=dart-http-boundary-2dFfnxL2f23_XmE3TBoxl9jl7KzoqjsvMQVXvA.Jbmo6wKZlqFj
    | user-agent: Dart pub 2.16.0-80.1.beta
IO  : HTTP response 302 Found for POST https://pub.mydomain.xy/api/packages/versions/newUpload
    | took 0:00:00.451820
    | location: https://pub.mydomain.xy/api/packages/versions/newUploadFinish
    | date: Tue, 11 Jan 2022 22:16:19 GMT
    | content-length: 0
    | strict-transport-security: max-age=315360000; includeSubDomains; preload
    | x-frame-options: DENY
    | content-type: text/plain; charset=utf-8
    | x-xss-protection: 1; mode=block
    | x-content-type-options: nosniff
    | server: dart:io with Shelf
IO  : HTTP GET https://pub.mydomain.xy/api/packages/versions/newUploadFinish
    | Accept: application/vnd.pub.v2+json
    | authorization: <censored>
    | user-agent: Dart pub 2.16.0-80.1.beta
IO  : HTTP response 200 OK for GET https://pub.mydomain.xy/api/packages/versions/newUploadFinish
    | took 0:00:00.053627
    | access-control-allow-origin: *
    | date: Tue, 11 Jan 2022 22:16:19 GMT
    | content-length: 56
    | strict-transport-security: max-age=315360000; includeSubDomains; preload
    | x-frame-options: DENY
    | content-type: application/json
    | x-xss-protection: 1; mode=block
    | x-content-type-options: nosniff
    | server: dart:io with Shelf
MSG : Successfully uploaded package.
FINE: Uploading finished (0.606s).

But I'm not sure if this is pub client, pub server or unpub issue, because I don't know what should actually happen:

• unpub should send 302 and the pub client should check it using the pub.dev token (or no token at all?) (this would make sense so I don't have overlapping packages with upstream)
• unpub should send 404 and the pub client would assume the package does not exist
• pub server should not return missing Authorization header error

EDIT: I missed this unpub PR, where they disable upstream check and return 404. But I'm still curious about the idea of preventing overlapping packages...

[sigurdm]

As I understand it the bug here is that we treat the 401 after a redirect as belonging to the redirecting server, and removes the invalid credentials belonging to that, even though we should be looking at the credentials for the redirectee.

I think that sounds very sensible.

[leoshusar]

Yeah, that's pretty much it.

And when the pub client receives redirect to another.pub.xy, it should look up and send auth token for this one, right?

[jonasfj]

it should look up and send auth token for this one, right?
No,

Only if it was redirected there from another.pub.xy.

[jonasfj]

I tried to check GET https://pub.dev/api/packages/oauth2 manually with my token and got a HTML response with this, is this even right?

Error <code>MissingAuthentication</code> <a href="#error-missingauthentication" class="hash-link">#</a></h1>
<p>Authentication is required, please add <code>authorization</code> header.

Maybe the error message could be better... but you're contacting pub.dev with a token that isn't valid. Authentication failed.

An error code is correct, maybe it should say "authentication failed" instead, make a reproducible example, and file an issue at https://github.com/dart-lang/pub-dev

[jonasfj]

I don't think we can help with unpub issues. A specification for writing custom pub servers is available here:
https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md

If you have any questions or concerns regarding it, please feel free to open an issue.

[jonasfj]

Or ping me in #hackers-pub-dev-📦 https://github.com/flutter/flutter/wiki/Chat

[jonasfj]

Filed dart-lang/pub-dev#5436 for a prettier message from pub.dev

[sigurdm]

here I see in my server logs 302, so it must've tried to check the official pub.dev

Reading the source around

pub/lib/src/authentication/client.dart

Line 46 in 1e78c68

_credential!.canAuthenticate(request.url.toString())) {

the client should not send an authentication header. Therefore the 401 should not be coming from pub.dev.