Plugin installation doesn't appear to respect proxies

[kyle-blair]

Summary

The sf cli cannot get through a proxy to install plugins.

Steps To Reproduce

No repo is necessary. You must be behind a restrictive web proxy to reproduce this issue. I use nvm, but I don't think that's part of the issue.

1. Configure proxy environment variables HTTP_PROXY and HTTPS_PROXY per the documentation.
2. Install node.
3. Run npm install --global @salesforce/cli.
4. Run a command that requires a just-in-time plugin such as sf package installed list --target-org yourTargetOrgAlias

Expected result

The required plugin is installed successfully and the command runs.

Actual result

$ sf package installed list --target-org yourTargetOrgAlias
Installing plugin [email protected]... ⣯ info There appears to be trouble with your network connection. Retrying...
Installing plugin [email protected]... failed
    JitPluginInstallError: Could not install @salesforce/plugin-packaging
    Code: JitPluginInstallError

System Information

$ echo $SHELL
/bin/bash
$ /bin/bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin22)
Copyright (C) 2007 Free Software Foundation, Inc.
$ echo "$BASH_VERSION"
3.2.57(1)-release

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.14.6",
  "nodeVersion": "node-v18.17.1",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.10 (core)",
    "@oclif/plugin-commands 3.0.3 (core)",
    "@oclif/plugin-help 6.0.3 (core)",
    "@oclif/plugin-not-found 3.0.2 (core)",
    "@oclif/plugin-plugins 3.9.2 (core)",
    "@oclif/plugin-search 1.0.3 (core)",
    "@oclif/plugin-update 4.1.2 (core)",
    "@oclif/plugin-version 2.0.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.1 (core)",
    "@oclif/plugin-which 3.0.5 (core)",
    "@salesforce/cli 2.14.6 (core)",
    "apex 2.3.20 (core)",
    "auth 2.8.22 (core)",
    "data 2.5.19 (core)",
    "deploy-retrieve 1.19.2 (core)",
    "info 2.6.49 (core)",
    "limits 2.3.39 (core)",
    "login 1.2.37 (core)",
    "marketplace 0.3.0 (core)",
    "org 2.11.4 (core)",
    "schema 2.3.30 (core)",
    "settings 1.4.35 (core)",
    "sobject 0.2.13 (core)",
    "source 2.10.43 (core)",
    "telemetry 2.3.6 (core)",
    "templates 55.5.14 (core)",
    "trust 2.6.20 (core)",
    "user 2.3.37 (core)",
    "sfdmu 4.30.0 (user)"
  ]
}

Additional information

I spent a while trying to figure out if I could get this to work somehow, with no luck. Many sources indicated it should "just work", such as yarn inheriting proxy configuration from npm. I was unable to set yarn configuration directly due to unrelated issues with corepack, so I'm out of things to try at this point.

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[cristiand391]

Hey @kyle-blair I'been testing this with a local proxy server and see the requests go through the proxy with the HTTPS_PROXY env var set (and nothing on npm/yarn config files);

HTTPS_PROXY='http://localhost:8888' sf package list -v na40:

are you able to check the proxy logs to verify there was a connection?

[kyle-blair]

Apologies, I didn't provide the full picture for the sake of simplicity but those details may be relevant. We use a private package registry as well and that traffic should not go through the proxy. I have additional proxy variables set, including NO_PROXY/no_proxy. I can see the internal registry is being picked up correctly (presumably from npm config), however it throws a 502 gateway error. Relevant dev debug output below. It seems like it's not picking up the no proxy variables, sending traffic to our internal registry through the proxy, and therefore the proxy returns 502. I'll try to check with our network team to see if something is going on on our side. I've always had issues installing sf/sfdx plugins even before this just in time behavior, but in those cases I'd see a 407 error.

 @oclif/plugins installing plugin @salesforce/plugin-packaging +0ms
  cli:yarn /Users/user/.local/share/sf: /Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli/node_modules/yarn/bin/yarn.js add @salesforce/[email protected] --non-intInstalling plugin [email protected]... ⡿ info There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://private.registry.com/@salesforce%2fplugin-packaging: tunneling socket could not be established, statusCode=502".
  cli:yarn yarn error Error: /Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli/node_modules/yarn/bin/yarn.js add @salesforce/[email protected] --non-interactive --mutex=file:/Users/user/.local/share/sf/yarn.lock --preferred-cache-folder=/Users/user/Library/Caches/sf/yarn --check-files --registry=https://private.registry.com/ exited with code 1
    at ChildProcess.<anonymous> (/Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli/node_modules/@oclif/plugin-plugins/lib/yarn.js:32:28)
    at ChildProcess.emit (node:events:514:28)
    at ChildProcess._handle.onexit (node:internal/child_process:291:12) +3m
  @oclif/plugins error installing plugin: Error: /Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli/node_modules/yarn/bin/yarn.js add @salesforce/[email protected] --non-interactive --mutex=file:/Users/user/.local/share/sf/yarn.lock --preferred-cache-folder=/Users/user/Library/Caches/sf/yarn --check-files --registry=https://private.registry.com/ exited with code 1
    at ChildProcess.<anonymous> (/Users/user/.nvm/versions/node/v18.17.1/lib/node_modules/@salesforce/cli/node_modules/@oclif/plugin-plugins/lib/yarn.js:32:28)
    at ChildProcess.emit (node:events:514:28)
    at ChildProcess._handle.onexit (node:internal/child_process:291:12) +3m

[cristiand391]

@kyle-blair looks like this is a know issue (and yarn v1 is in maintenance mode so we don't expect it to be fixed) but someone found a workaround using npm/yarn config files:

yarnpkg/yarn#5048 (comment)

[kyle-blair]

Initial testing of that workaround does not resolve the issue. I was hopeful since there were a few differences with how my proxy has been configured (for years): I didn't have a PROXY environment variable, and I use some variations of the specified config keys for the npm settings. I'll try a few more things, however, given that issue is 6 years old and the workaround is 3 years old, I think there are too many unknowns or things that could have changed in that time.

I assume a lot of Salesforce's customers use corporate proxies and (maybe less so) internal package registries. I'm surprised this wasn't a factor in the decision to use just in time plugins, let alone tested. It's another item in the long list of painful experiences with the cli.

When was just in time plugin installation introduced? My best bet at this point may be to go back to that version for a while.

[kyle-blair]

I did a "clean slate" attempt at the workaround, and the just in time plugin install worked this time. Here's what I did:

1. Opened a new terminal.
2. Cleared all existing proxy environment variables. unset HTTP_PROXY HTTPS_PROXY # etc ...
3. Deleted all existing npm proxy config. npm config delete proxy (this alters .npmrc, so back it up)
4. Deleted all existing yarn proxy config.
5. Followed the exact steps documented here. I put the npm settings in "$HOME"/.npmrc and copied the file to my current working project directory just to be thorough. I did the same for yarn configuration in .yarnrc; one file in the home directory and a copy in the current working directory.

The kicker is, after the plugin was installed, the command failed because the proxy wasn't configured correctly. So, this is all a one-time manual setup needed to install just in time plugins and then you need to revert back to your typical proxy configuration to use the cli normally. I wouldn't consider that a viable solution, and it negates any advantages to just in time plugin installation.

[kyle-blair]

So, what is the plan here? This is a significant issue for anyone behind a corporate proxy with a private npm registry and the workaround is not reasonable.

[cristiand391]

The kicker is, after the plugin was installed, the command failed because the proxy wasn't configured correctly

@kyle-blair so you got an error when running the packaging command? Can you share the exact error?
CLI commands uses different libs respect proxy env vars (not npm/yarn config), maybe we missed one in packaging.

[kyle-blair]

As explained in my last comment, the workaround you shared doesn't work when done in addition to the typical, required proxy variables. I had to start with no variables set and then set only the variables mentioned in the workaround to be able to install the plugin. That configuration is not valid for everyday proxy use. Therefore, I have to follow those exact steps in order to install (and presumably update) plugins, and then revert back to my typical configuration.

The original issue still stands. Just in time plugin installation does not work with a corporate proxy and private npm registry. The workaround is just that--a workaround--not a viable solution.

As far as I'm concerned, just in time plugin installation is not worth the trouble. I'm not sure what problems it was intended to address, but I'd just as soon turn it off if I could.

[hudec117]

Facing this in a corporate environment too and it's making it impossible to use.

[mmittereder]

@cristiand391
This is also a huge issue for my team as well. We have a significate cooperate presence in using Salesforce and are requesting an ETA on when this issue will be officially resolved by Salesforce.

[mitchspano]

This is a big issue for us as well.

[hudec117]

Potential Workaround: Downgrade to 2.3.8 (or a version close to) using https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_setup_install_cli.htm#sfdx_setup_install_cli_olderversions and after running sf package version create a few times, it does get past the yarn/network errors and installs the packaging plugin.

[Rogeriohsjr]

I got this error as well.
I uninstalled all the sf and sfdx that I have in my machine.
I installed only sf cli by npm, I also try executable file, and I wasn't able to overcome this issue.

The solution described on this thread worked for me:
https://salesforce.stackexchange.com/a/402568

1. npm install @salesforce/cli -g
2. npm install @salesforce/plugin-packaging -g
3. link the plugin

I had to run that into a bash terminal

Owner@XXX MINGW64 ~/AppData/Roaming/npm/node_modules/@salesforce/plugin-packaging
$ sf plugins link
@salesforce/cli: linking plugin @salesforce/plugin-packaging... done

The odd part is that if I run that on terminal, I get the error below:

C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\plugin-packaging>sf plugins link
@salesforce/cli: linking plugin @salesforce/plugin-packaging... !
    Error: C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\cli\node_modules\yarn\bin\yarn.js --non-interactive 
    --mutex=file:C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\plugin-packaging\yarn.lock
    --preferred-cache-folder=C:\Users\Owner\AppData\Local\sf\yarn --check-files --silent exited with code 1

That worked for me, In my case I was generating a new package version.

VERSION
@salesforce/cli/2.16.7 win32-x64 node-v18.14.2

[mshanemc]

Longer-term, we're going to come up with a solid solution for this in oclif. But there's some complexity there.

In the short-term, we've converted plugin-packaging from JIT to bundled, which should help unblock the majority of people who have posted here.

It won't solve the installation of plugins from private registries from behind proxies use case (that'll be the longer term fix).

[Saifalkayali]

Longer-term, we're going to come up with a solid solution for this in oclif. But there's some complexity there.

In the short-term, we've converted plugin-packaging from JIT to bundled, which should help unblock the majority of people who have posted here.

It won't solve the installation of plugins from private registries from behind proxies use case (that'll be the longer term fix).

This actually now resolves my issue of not being able to install the packaging plugin (that is required for sf package install command) due to using an internal company NPM registry rather than the public/external NPM registry, thanks for the fix. I believe that now the plugin is being installed within the SF CLI package, it's no longer failing upon package installs.

Note: I did have to specify the CLI version to use rc since it's not officially released yet but will be soon!

'2.19.7 (Nov 29, 2023) [stable-rc]'

[git2gus]

This issue has been linked to a new work item: W-14581559

[pavanbhat1999]

I got this error as well. I uninstalled all the sf and sfdx that I have in my machine. I installed only sf cli by npm, I also try executable file, and I wasn't able to overcome this issue.

The solution described on this thread worked for me: https://salesforce.stackexchange.com/a/402568

1. npm install @salesforce/cli -g
2. npm install @salesforce/plugin-packaging -g
3. link the plugin

I had to run that into a bash terminal

Owner@XXX MINGW64 ~/AppData/Roaming/npm/node_modules/@salesforce/plugin-packaging
$ sf plugins link
@salesforce/cli: linking plugin @salesforce/plugin-packaging... done

The odd part is that if I run that on terminal, I get the error below:

C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\plugin-packaging>sf plugins link
@salesforce/cli: linking plugin @salesforce/plugin-packaging... !
    Error: C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\cli\node_modules\yarn\bin\yarn.js --non-interactive 
    --mutex=file:C:\Users\Owner\AppData\Roaming\npm\node_modules\@salesforce\plugin-packaging\yarn.lock
    --preferred-cache-folder=C:\Users\Owner\AppData\Local\sf\yarn --check-files --silent exited with code 1

That worked for me, In my case I was generating a new package version.

VERSION @salesforce/cli/2.16.7 win32-x64 node-v18.14.2

in bash also i am getting the same error

[mshanemc]

All I've got is an update on this. The plugin installation is part of the oclif framework. (oclif/plugin-plugins) is a PR to switch from using yarn1 to using current npm for managing all the installed plugins. It's obviously not trivial to change and test, and there'll probably be some required steps (yarnrc vs. npmrc, maybe some envs)

But the good news is that current npm should respect all your proxy stuff better than the old yarn does.

[mdonnalley]

@kyle-blair Have you seen this announcement? We're switching to npm to execute plugin installs, which should handle proxies better than yarn v1. It'd be super helpful if you could give it a test run to see if it resolves your issue.

You can access it via the qa version (sf update qa or npm install -g @salesforce/cli@qa)

[kyle-blair]

I am on leave for a while, but let me pass this along to a colleague.