Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📌 Check in package-lock.json as well #1198

Merged
merged 2 commits into from
Feb 16, 2025

Conversation

shankari
Copy link
Contributor

So we can have the CI use known good versions of the dependencies. Without this change, the underlying libraries can change based on the loosely defined semver in the package.json, which can cause unexpected failures.

An example of such a failure is:
e-mission/e-mission-docs#1113

Copying over the package-lock.json overrides package.json and forces the build to use the exact versions specified e-mission/e-mission-docs#1113 (comment)

Note that package-lock.json is apparently designed to checked in to source control. It used to be checked in (thanks to a contribution from the UNSW team) but I removed it because every run of npm install would override it, resulting in confusing merge conflicts.

However, it looks like npm has now changed its behavior so it respects the package-lock.json if it exists.
e-mission/e-mission-docs#1113 (comment)

So we restore it by copying in known good versions - from https://github.com/e-mission/e-mission-phone/actions/runs/12448538357 for the "serve" case and from the last successful internal build of NREL OpenPATH for the cordovabuild case.

We also change bin/configure_xml_and_json.js to copy these files over as well

Testing done:

  • Copying package-lock.json into the repo causes npm install to work, and the tests to pass
  • Testing of the setup changes will happen through CI

@shankari
Copy link
Contributor Author

Ah, it looks like copying the package-lock.json first doesn't work; we need to copy it after the first npm install and then run npm install again. I guess e-mission/e-mission-docs#1113 (comment) worked because I ran the steps with an existing e-mission-phone directory.

So we can have the CI use known good versions of the dependencies.
Without this change, the underlying libraries can change based on the loosely
defined semver in the `package.json`, which can cause unexpected failures.

An example of such a failure is:
e-mission/e-mission-docs#1113

Copying over the `package-lock.json` overrides `package.json` and forces the build to use the exact versions specified
e-mission/e-mission-docs#1113 (comment)

Note that `package-lock.json` is apparently designed to checked in to source
control. It used to be checked in (thanks to a contribution from the UNSW team)
but I removed it because every run of `npm install` would override it,
resulting in confusing merge conflicts.

However, it looks like `npm` has now changed its behavior so it respects the
`package-lock.json` if it exists.
e-mission/e-mission-docs#1113 (comment)

So we restore it by copying in known good versions - from
https://github.com/e-mission/e-mission-phone/actions/runs/12448538357
for the "serve" case and from the last successful internal build of NREL
OpenPATH for the `cordovabuild` case.

We also change `bin/configure_xml_and_json.js` to copy these files over as well

Testing done:
- Copying `package-lock.json` into the repo causes `npm install` to work, and
  the tests to pass
- Testing of the setup changes will happen through CI
@shankari shankari force-pushed the force_package_dependencies_fix_tests branch from a278bb0 to be94357 Compare February 16, 2025 21:57
@shankari
Copy link
Contributor Author

So this is a bit weird. In the first round of this, I checked in the package-lock.json that was in thee-mission-phone directory after running npm install. However, while using that version, I consistently got the error

Copied config.serve.xml -> config.xml and package.serve.json -> package.json package-lock.serve.json -> package-lock.json
Re-run npm install to install the locked packages
npm WARN ancient lockfile 
npm WARN ancient lockfile The package-lock.json file was created with an old version of npm,
npm WARN ancient lockfile so supplemental metadata must be fetched from the registry.
npm WARN ancient lockfile 
npm WARN ancient lockfile This is a one-time fix-up, please be patient...
npm WARN ancient lockfile 
npm WARN ancient lockfile HttpErrorGeneral: 404 Not Found - GET https://registry.npmjs.org/e-mission-common - Not found
npm WARN ancient lockfile     at /Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/npm-registry-fetch/lib/check-response.js:95:15
npm WARN ancient lockfile     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
npm WARN ancient lockfile     at async RegistryFetcher.packument (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/pacote/lib/registry.js:87:19)
npm WARN ancient lockfile     at async RegistryFetcher.manifest (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/pacote/lib/registry.js:118:23)
npm WARN ancient lockfile     at async Array.<anonymous> (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:727:24)
npm WARN ancient lockfile  Could not fetch metadata for e-mission-common@ HttpErrorGeneral: 404 Not Found - GET https://registry.npmjs.org/e-mission-common - Not found
npm WARN ancient lockfile     at /Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/npm-registry-fetch/lib/check-response.js:95:15
npm WARN ancient lockfile     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
npm WARN ancient lockfile     at async RegistryFetcher.packument (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/pacote/lib/registry.js:87:19)
npm WARN ancient lockfile     at async RegistryFetcher.manifest (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/pacote/lib/registry.js:118:23)
npm WARN ancient lockfile     at async Array.<anonymous> (/Users/kshankar/.nvm/versions/node/v20.9.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:727:24) {

But using the version that was uploaded from https://github.com/e-mission/e-mission-phone/actions/runs/12448538357
does not generate that error.

Although we tried to override the `package-lock.json`, it is overridden, in
turn by npm when we run npm install after changing `package.json`

And because we have a `package.serve.json` and a `configure_xml_and_js` script,
we change the `package.json` every time we clone, so the `package-lock.json`
never stick.
#1198 (comment)

The fix is to re-copy just the lock file after the first `npm install` and then
run `npm install` again, essentially duplicating the flow in
#1198 (comment)
@shankari
Copy link
Contributor Author

So this is a bit weird. In the first round of this, I checked in the package-lock.json that was in the e-mission-phone directory after running npm install. However, while using that version, I consistently got the error

Ah I think I have more clarity - the package-lock.json generated from the original package.json displays the error.
And it is generated every time that package.json changes, but not at other times.

So the flow needs to be:

  • run the configure_xml_and_json script; this changes package.json
  • npm install re-writes package-lock.json
  • copy over the working package-lock.json and only the working package-lock.json
  • re-run npm install

Copy link

codecov bot commented Feb 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 29.78%. Comparing base (e9abaee) to head (1089679).
Report is 7 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1198   +/-   ##
=======================================
  Coverage   29.78%   29.78%           
=======================================
  Files         123      123           
  Lines        4955     4955           
  Branches     1142     1095   -47     
=======================================
  Hits         1476     1476           
- Misses       3477     3479    +2     
+ Partials        2        0    -2     
Flag Coverage Δ
unit 29.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes

@shankari shankari merged commit 9a6ae19 into master Feb 16, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Tasks completed
Development

Successfully merging this pull request may close these issues.

1 participant