Migrate Perseus to pnpm #2202

jeremywiebe · 2025-02-05T22:56:19Z

Summary:

This PR migrates this repo from yarn to pnpm. The version of Yarn we are using is very outdated and is no longer supported. Khan Academy has made an engineering-wide decision to adopt pnpm as its "package manager of choice" and so let's adopt!

Notable changes:

Use workspace: syntax for in-repo dependencies
Use catalog: syntax for peer/dev dependencies (easier "global" upgrade of these) - and Github just added support for catalog: support Dependabot yesterday!!
Build system is simplified because we don't have to worry about build order

Issue: LEMS-2825

Test plan:

pnpm build
pnpm dev
pnpm start # storybook
pnpm cypress:ci

github-actions · 2025-02-05T23:46:28Z

Size Change: -604 kB (-41.08%) 🎉

Total Size: 866 kB

Filename	Size	Change
`packages/kas/dist/es/index.js`	38.9 kB	-88 B (-0.23%)
`packages/kmath/dist/es/index.js`	10.4 kB	-76.4 kB (-87.99%)	🏆
`packages/math-input/dist/es/index.js`	77.6 kB	-135 B (-0.17%)
`packages/perseus-core/dist/es/index.js`	29.6 kB	-18.6 kB (-38.56%)	🎉
`packages/perseus-editor/dist/es/index.js`	275 kB	-414 kB (-60.05%)	🏆
`packages/perseus-linter/dist/es/index.js`	22 kB	-114 B (-0.52%)
`packages/perseus-score/dist/es/index.js`	20.3 kB	-94.8 kB (-82.33%)	🏆
`packages/perseus/dist/es/index.js`	367 kB	-270 B (-0.07%)
`packages/pure-markdown/dist/es/index.js`	3.54 kB	-119 B (-3.25%)
`packages/simple-markdown/dist/es/index.js`	12.5 kB	-1 B (-0.01%)

ℹ️ View Unchanged

Filename	Size
`packages/keypad-context/dist/es/index.js`	760 B
`packages/math-input/dist/es/strings.js`	1.79 kB
`packages/perseus/dist/es/strings.js`	6.57 kB

_{compressed-size-action}

mark-fitzgerald

Storybook/Vite/Aphrodite adjustment looks good. Thanks for keeping it along for the configuration changes.

github-actions · 2025-02-06T00:52:53Z

npm Snapshot: Published

Good news!! We've packaged up the latest commit from this PR (e2ad2ea) and published it to npm. You
can install it using the tag PR2202.

Example:

pnpm add @khanacademy/perseus@PR2202

If you are working in Khan Academy's webapp, you can run:

./dev/tools/bump_perseus_version.sh -t PR2202

jeremywiebe · 2025-02-06T23:44:27Z

config/build/rollup.config.js

@@ -34,74 +32,6 @@ const rootDir = ancesdir(__dirname);
 *               from the default behavior.
 */

-// Kahn's algorithm
-// https://en.wikipedia.org/wiki/Topological_sorting#Kahn%27s_algorithm
-const topoSort = (yarnWorkspacesOutput) => {


Not needed. WonderBlocks doesn't build in any dependency order, and after thinking about it a bit, we don't need to either. The build doesn't depend on build artifacts of other in-repo packages, so this simplifies our build a bunch!

@kevinb-khan I tagged you mostly for these changes. I'm pretty sure we don't need to build in any dependency order.

Does it still build if you reverse the topological sort? If it does, I think we're probably okay.

Yep. I did a manual ordering in the reverse of what topoSort() produces and it built fine.

jeremywiebe · 2025-02-06T23:46:08Z

config/build/rollup.config.js

-                    platform === "browser" && format === "esm"
-                        ? "runtime"
-                        : "bundled",
+                babelHelpers: "runtime",


WonderBlocks uses runtime for both ES and CJS builds, so we'll do the same!

https://www.npmjs.com/package/@rollup/plugin-babel#babelhelpers

jeremywiebe · 2025-02-06T23:48:53Z

vite.config.ts

@@ -0,0 +1,61 @@
+import {readFileSync} from "node:fs";


Most of this file is from the file that was moved from dev/vite.config.ts

jeremywiebe · 2025-02-06T23:49:11Z

vendor/hubble/package.json

@@ -1,5 +1,6 @@
 {
  "name": "hubble",
+  "private": true,


Juuuuust so we don't try to publish this package!

jeremywiebe · 2025-02-07T17:25:25Z

.github/workflows/node-ci.yml

+            # MUST be before we install node_modules as that depends on finding
+            # Cypress binaries in this cache!
+            - uses: actions/cache@v4
+              with:
+                  path: ~/.cache/Cypress
+                  key: cypress-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
+


Cypress runs a post-install script (which we've allowed in package.json), but it doesn't seem to include downloading the Cypress binaries if we've cached node_modules (next step). By caching this directory, the post-install script is able to find a cached binary and connect it.

On a clean install (or Cypress upgrade) the node_modules cache will be busted and the cypress npm package properly downloads the correct binary and caches it in ~/.cache/Cypress.

That's nice, I imagine that would speed things up a bit.

On a random sampling of one other PR, this workflow took 1m 34s.

On this PR, this same workflow took 1m 10s.

So, yes, 24s faster!

jeremywiebe · 2025-02-07T17:26:42Z

.storybook/main.ts

@@ -1,4 +1,4 @@
-import viteConfig from "../dev/vite.config";


I moved some vite configs around. It always felt weird that we used dev/vite.config.ts as our base. Now we have a shared base config in the root of the repo and both Storybook and dev/ leverage it.

This will pave the way in the future for us to use Vitest for testing (faster!!)

jeremywiebe · 2025-02-07T17:27:02Z

.storybook/main.ts

@@ -72,7 +72,7 @@ const config: StorybookConfig = {
            },
            // Fix from: https://github.com/storybookjs/storybook/issues/25256#issuecomment-1866441206
            assetsInclude: ["/sb-preview/runtime.js"],
-            plugins: [...viteConfig.plugins, lessWrapper],
+            plugins: [...(viteConfig.plugins ?? []), lessWrapper],


No guarantee that there are plugins defined in the base config.

jeremywiebe · 2025-02-07T17:27:38Z

.storybook/preview.tsx

@@ -1,9 +1,11 @@
 import * as React from "react";
 import {color} from "@khanacademy/wonder-blocks-tokens";
 import {RenderStateRoot} from "@khanacademy/wonder-blocks-core";
-import {Dependencies} from "@khanacademy/perseus";


The root package doesn't depend on one of its monorepo packages, so this needed to change to a relative path now that we're using pnpm.

By "one of its monorepo packages," you mean @khanacademy/perseus? I think the resolutions for these monorepo package imports are defined in vite.config.ts. Maybe there's a problem with how that's being done?

By "one of its monorepo packages," you mean @khanacademy/perseus?

Yes

I think the resolutions for these monorepo package imports are defined in vite.config.ts. Maybe there's a problem with how that's being done?

We did have aliases. But with pnpm, we don't need them anymore as pnpm correctly sets up in-repo dependencies. I removed the aliases from the Vite config and this was the only thing that needed adjustment. I figured it was cleaner to have fewer things (and customizations) in Vite config if pnpm solves the same problem. Changing this to a relative path import worked (we were already importing the DependenciesContext using a relative path in this file).

jeremywiebe · 2025-02-07T17:42:51Z

config/build/create-babel-plugins.js

@@ -8,15 +8,13 @@ module.exports = function createBabelPlugins({platform, format, coverage}) {
    if (coverage) {
        plugins.push("istanbul");
    }
-    if (platform === "browser" && format === "esm") {


I switched this because we always build for "browser" and I made a change in rollup-config.js to match how WB builds - namely, we always use babelHelpers: "runtime" and so we need this plugin always.

jeremywiebe · 2025-02-07T19:46:26Z

config/test/test.config.js

@@ -50,7 +50,7 @@ module.exports = {
    },
    // Allow transforming files imported from @phosphor-icons/core.
    // This is required by the .svg transform above.
-    transformIgnorePatterns: ["/node_modules/(?!(@phosphor-icons/core)/)"],
+    transformIgnorePatterns: ["/node_modules/.pnpm/(?!@phosphor-icons.core@)"],


Shamelessly stole this also from WB. pnpm organizes the details of node_modules differently!

jeremywiebe · 2025-02-07T19:48:05Z

dev/package.json

+        "@khanacademy/simple-markdown": "workspace:*",
+        "@khanacademy/wonder-blocks-banner": "catalog:",


workspace:* for in-repo dependencies, catalog: for the deps that we'll be provided at runtime (peer deps).

This should make managing peer and dev dependency versions so much easier! Plus, Dependabot (Github) just announced full support for pnpm catalogs a few days ago!

jeremywiebe · 2025-02-07T19:48:39Z

package.json

@@ -55,7 +55,6 @@
        "@testing-library/jest-dom": "^6.4.6",
        "@testing-library/react": "^16.0.0",
        "@testing-library/user-event": "^14.5.2",
-        "@types/classnames": "^2.3.1",


pnpm kept telling me that this library ships with its types built-in now... so we don't need this dep.

jeremywiebe · 2025-02-07T19:49:00Z

package.json

@@ -123,22 +125,24 @@
        "tiny-invariant": "^1.3.1",
        "typescript": "5.7.2",
        "typescript-coverage-report": "^0.7.0",
+        "vite": "5.1.0",


Moved up from dev/

jeremywiebe · 2025-02-07T19:49:12Z

package.json

        "vite-plugin-istanbul": "^5.0.0",
        "winston": "^3.7.2"
    },
    "scripts": {
-        "gen:parsers": "yarn --cwd packages/kas gen:parsers",
-        "prebuild": "yarn gen:parsers",
+        "preinstall": "npx -y only-allow pnpm",


This catches if someone runs yarn...

jeremywiebe · 2025-02-07T19:50:07Z

package.json

-    "dependencies": {}
+    "packageManager": "[email protected]+sha512.b8fef5494bd3fe4cbd4edabd0745df2ee5be3e4b0b8b08fa643aa3e4c6702ccc0f00d68fa8a8c9858a735a0032485a44990ed2810526c875e416f001b17df12b",
+    "pnpm": {
+        "onlyBuiltDependencies": [


This flags packages that we explicitly allow to run post-install scripts during pnpm install. A nice safety feature to protect us from supply-chain attacks.

jeremywiebe · 2025-02-07T19:51:19Z

packages/kas/package.json

@@ -23,19 +23,18 @@
    ],
    "scripts": {
        "prepublishOnly": "../../utils/package-pre-publish-check.sh",
-        "gen:parsers": "node src/parser-generator.ts",
-        "test": "bash -c 'yarn --silent --cwd \"../..\" test ${@:0} $($([[ ${@: -1} = -* ]] || [[ ${@: -1} = bash ]]) && echo $PWD)'"


I removed this. As far as I know, nobody cd's into the various package directories and runs tests. It's just as easy to run from the root, and you can just as easily run pnpm test packages/kas from the root!

jeremywiebe · 2025-02-07T19:52:07Z

packages/math-input/less/main.less

@@ -1,2 +1,2 @@
 @import "./overrides.less";
-@import (inline) "../../../node_modules/mathquill/build/mathquill.css";
+@import (inline) "../node_modules/mathquill/build/mathquill.css";


pnpm installs direct package dependencies into the package's local node_modules folder, so these relative paths now make more sense.

pnpm-workspace.yaml

jeremywiebe · 2025-02-07T23:10:26Z

Size Change: -191 kB (-12.89%) 👏

Total Size: 1.29 MB

Filename Size Change
packages/kas/dist/es/index.js 38.9 kB -88 B (-0.23%)
packages/kmath/dist/es/index.js 10.4 kB -76.4 kB (-87.99%) 🏆
packages/math-input/dist/es/index.js 77.5 kB -141 B (-0.18%)
packages/perseus-core/dist/es/index.js 27.7 kB -18.6 kB (-40.12%) 🎉
packages/perseus-editor/dist/es/index.js 688 kB -158 B (-0.02%)
packages/perseus-linter/dist/es/index.js 22.1 kB -114 B (-0.51%)
packages/perseus-score/dist/es/index.js 20.1 kB -94.8 kB (-82.5%) 🏆
packages/perseus/dist/es/index.js 379 kB -273 B (-0.07%)
packages/pure-markdown/dist/es/index.js 3.54 kB -118 B (-3.23%)
ℹ️ View Unchanged
compressed-size-action

I checked these size changes. We were bundling jQuery which we've marked as a dev/peer dep (meaning that we expect the host app to supply it). We no longer are. There are a few reductions in that we're using babelHelpers: "runtime" to allow us to use the host app's babel (which is what Wonder Blocks also does).

…ow that we're always setting babelHelpers to 'runtime'

Co-authored-by: Ben Christel <[email protected]>

…ok/main.ts

…sets

jeremywiebe self-assigned this Feb 5, 2025

jeremywiebe force-pushed the jer/pnpm branch from b5bfb41 to 218c35f Compare February 5, 2025 23:12

mark-fitzgerald approved these changes Feb 5, 2025

View reviewed changes

jeremywiebe commented Feb 6, 2025

View reviewed changes

jeremywiebe force-pushed the jer/pnpm branch from d0302a7 to 593e25c Compare February 7, 2025 00:56

jeremywiebe commented Feb 7, 2025

View reviewed changes

pnpm-workspace.yaml Show resolved Hide resolved

jeremywiebe marked this pull request as ready for review February 7, 2025 23:10

jeremywiebe requested review from mark-fitzgerald, jeresig, jandrade and kevinb-khan February 7, 2025 23:11

jeremywiebe and others added 23 commits February 19, 2025 14:04

Make all packages in teh repo that aren't publishable PRIVATE

25742a6

Allow postinstall scripts for tooling we use

177a595

Restore aliases for vendor/ packages

a56664a

Change build to always set babelHelpers to 'runtime'

f972040

Remove console.log()

4082f90

I can't make up my mind!

fa97581

Ensure babel plugins always include @babel/plugin-transform-runtime n…

5adc7b4

…ow that we're always setting babelHelpers to 'runtime'

Ensure we get the right version of pnpm

9b91e3e

I was wrong

1ffb362

Looks like action can find lockfile itself

9ab3f4d

Trying to get Cypress to work

42933fb

Manage Cypress cache ourselves

ebc67b6

prettier

4a16bda

Remove unused action param

86c21fc

Fix utils/sync-dependencies.ts to update pnpm-workspace.yaml

c94bb2d

Update .github/workflows/node-ci.yml

fb94783

Co-authored-by: Ben Christel <[email protected]>

Revert eslint change and just suppress the one rule violation

6895c6d

remove unsupported -s from pnpm invocation

9982e74

Add Jira to Jest v30 upgrade task

95194f9

Smith the dependencies just a bit more

96a10c8

Move process.env.STORYBOOK setting to dev/vite.config.ts and .storybo…

6a9136b

…ok/main.ts

Just use 'true'

d61d78c

Sync wb versions

510d255

jeremywiebe force-pushed the jer/pnpm branch from 7def4f6 to 510d255 Compare February 19, 2025 22:43

jeremywiebe added 3 commits February 19, 2025 14:47

Don't use JSON.stringify() for simple values

801da1d

We need phosphor icons in root package.json as some tooling uses them

ae179e5

Don't try to restore/cache pnpm dependencies when checking for change…

e2ad2ea

…sets

jeremywiebe merged commit c7f6f63 into main Feb 20, 2025
8 checks passed

jeremywiebe deleted the jer/pnpm branch February 20, 2025 04:15

khan-actions-bot mentioned this pull request Feb 20, 2025

Version Packages #2258

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate Perseus to pnpm #2202

Migrate Perseus to pnpm #2202

jeremywiebe commented Feb 5, 2025 •

edited

Loading

github-actions bot commented Feb 5, 2025 •

edited

Loading

mark-fitzgerald left a comment

github-actions bot commented Feb 6, 2025 •

edited

Loading

jeremywiebe Feb 6, 2025

jeremywiebe Feb 7, 2025

benchristel Feb 8, 2025

jeremywiebe Feb 10, 2025

jeremywiebe Feb 6, 2025

jeremywiebe Feb 6, 2025

jeremywiebe Feb 6, 2025

jeremywiebe Feb 7, 2025

handeyeco Feb 10, 2025

jeremywiebe Feb 10, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

benchristel Feb 8, 2025

jeremywiebe Feb 10, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe Feb 7, 2025

jeremywiebe commented Feb 7, 2025

		"@khanacademy/simple-markdown": "workspace:*",
		"@khanacademy/wonder-blocks-banner": "catalog:",

Migrate Perseus to pnpm #2202

Migrate Perseus to pnpm #2202

Conversation

jeremywiebe commented Feb 5, 2025 • edited Loading

Summary:

Test plan:

github-actions bot commented Feb 5, 2025 • edited Loading

mark-fitzgerald left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 6, 2025 • edited Loading

npm Snapshot: Published

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremywiebe commented Feb 7, 2025

jeremywiebe commented Feb 5, 2025 •

edited

Loading

github-actions bot commented Feb 5, 2025 •

edited

Loading

github-actions bot commented Feb 6, 2025 •

edited

Loading