🚀 Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

IgorMinar · 2024-04-10T21:41:10Z

Describe the solution

NuxtHub, Nuxt, and Nitro would like to be able to hide / abstract away wrangler.toml configuration for full-stack app deployments via wrangler pages deploy and Pages CI.

In the ideal scenario, Nitro (powering Nuxt and NuxtHub) would as part of the build generate a config file (wrangler.toml or more likely a subset of it stored in a simpler format), and this configuration would be picked up by wrangler when the application is deployed to pages.

The configuration supported in this way is limited to JS runtime config (compat flags and date and bindings config). This would enable to frameworks to completely manage runtime configuration on behalf of the user without exposing them to often unnecessary deployment configuration.

Without any other changes, the only way to approximate this behavior now is by the framework generating wrangler.toml and writing it into the source directory of the project, which increases noise for developers and creates a potential for clobbering developer's modifications.

// @pi0 @atinux

The text was updated successfully, but these errors were encountered:

penalosa · 2024-04-10T22:23:26Z

This mostly works today, by generating a wrangler.toml file in the source directory of the project during the build. I'm not sure I'm quite clear on the specific ask here:

Increasing noise for developers: Frameworks can generate a dynamic wrangler.toml file during the build, which users would never see
Clobbering user-written wrangler.toml files: How are frameworks planning to reconcile a user-written wrangler.toml and the framework authored one? Is this for things like automatically adding a compat flag?

I wonder if wrangler.toml is the wrong abstraction for frameworks to use here, since it sounds like there's a need for much greater control over the output than users would want/need directly. Maybe supporting frameworks outputting a _worker.bundle file directly might be the way to go here? That would allow them to fully customise the wrangler.toml parsing and metadata construction behaviour, as well as completely bypass Wrangler's bundling & module rules.

pi0 · 2024-04-10T22:56:02Z

Supporting _worker.bundle would be fantastic for Nitro-based deployments.

IgorMinar · 2024-04-10T23:07:36Z

Increasing noise for developers: Frameworks can generate a dynamic wrangler.toml file during the build, which users would never see

@penalosa you'd see it if you were doing direct deploy rather than using the pages CI to deploy.

IgorMinar · 2024-04-10T23:13:30Z

Maybe supporting frameworks outputting a _worker.bundle file directly might be the way to go here?

That's what I'm thinking as well. The build output will then become environment specific (unlike today when it's environment-agnostic since _worker.bundle is created during deployment based on env selection), but that's ok.

pi0 · 2024-04-10T23:31:51Z

This mostly works today, by generating a wrangler.toml file in the source directory of the project during the build. I'm not sure I'm quite clear on the specific ask here:

Just made an experimental change to try this (nitrojs/nitro#2355) as a short-time hotfix.

Here is the simple repo for a nitro app: https://github.com/pi0/nitro-app-cf

If you want to try the CI locally, you can use pnpm install > CI=1 pnpm build and notice it will generate a top-level wrangler.toml file (it is dynamically generated by nitro)

From build logs, notice that No Pages config file found is thrown before build command.

build.log

2024-04-10T23:26:44.40083Z	Cloning repository...
2024-04-10T23:26:45.07849Z	From https://github.com/pi0/nitro-app-cf
2024-04-10T23:26:45.079085Z	 * branch            4aeb7435e3d33a368d2c18cf6a326bded14c048f -> FETCH_HEAD
2024-04-10T23:26:45.079195Z	
2024-04-10T23:26:45.108142Z	HEAD is now at 4aeb743 add node_compat
2024-04-10T23:26:45.108869Z	
2024-04-10T23:26:45.203127Z	
2024-04-10T23:26:45.20368Z	Using v2 root directory strategy
2024-04-10T23:26:45.229732Z	Success: Finished cloning repository files
2024-04-10T23:26:46.815361Z	
2024-04-10T23:26:46.902351Z	�[31m✘ �[41;31m[�[41;97mERROR�[41;31m]�[0m �[1mNo Pages config file found�[0m
2024-04-10T23:26:46.902616Z	
2024-04-10T23:26:46.902701Z	
2024-04-10T23:26:46.906712Z	🪵  Logs were written to "/root/.config/.wrangler/logs/wrangler-2024-04-10_23-26-46_607.log"
2024-04-10T23:26:46.917517Z	No wrangler.toml file found
2024-04-10T23:26:47.077817Z	Detected the following tools from environment: [email protected], [email protected]
2024-04-10T23:26:47.862979Z	Installing project dependencies: pnpm install
2024-04-10T23:26:48.570437Z	Lockfile is up to date, resolution step is skipped
2024-04-10T23:26:48.60017Z	Progress: resolved 1, reused 0, downloaded 0, added 0
2024-04-10T23:26:48.671753Z	Packages: +396
2024-04-10T23:26:48.672033Z	++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2024-04-10T23:26:49.601184Z	Progress: resolved 396, reused 0, downloaded 137, added 137
2024-04-10T23:26:50.604868Z	Progress: resolved 396, reused 0, downloaded 392, added 391
2024-04-10T23:26:50.768457Z	Progress: resolved 396, reused 0, downloaded 396, added 396, done
2024-04-10T23:26:50.971695Z	.../[email protected]/node_modules/esbuild postinstall$ node install.js
2024-04-10T23:26:51.083311Z	.../[email protected]/node_modules/esbuild postinstall: Done
2024-04-10T23:26:51.19905Z	
2024-04-10T23:26:51.199352Z	devDependencies:
2024-04-10T23:26:51.199573Z	+ nitropack <- nitropack-nightly 2.10.0-28546505.8e88dd41
2024-04-10T23:26:51.199723Z	
2024-04-10T23:26:51.204207Z	
2024-04-10T23:26:51.204433Z	> @ prepare /opt/buildhome/repo
2024-04-10T23:26:51.204553Z	> nitro prepare
2024-04-10T23:26:51.20467Z	
2024-04-10T23:26:52.053025Z	Done in 3.8s
2024-04-10T23:26:52.134518Z	Executing user command: npm run build
2024-04-10T23:26:52.932187Z	
2024-04-10T23:26:52.932464Z	> build
2024-04-10T23:26:52.932676Z	> nitro build
2024-04-10T23:26:52.932811Z	
2024-04-10T23:26:53.65194Z	[success] [nitro] Generated public dist
2024-04-10T23:26:53.680071Z	[info] [nitro] Building Nitro Server (preset: `cloudflare-pages`)
2024-04-10T23:26:54.820655Z	[success] [nitro] Nitro Server built
2024-04-10T23:26:54.849683Z	  ├─ dist/_worker.js/chunks/routes/index.mjs (745 B) (380 B gzip)
2024-04-10T23:26:54.849938Z	  ├─ dist/_worker.js/chunks/routes/index.mjs.map (818 B) (294 B gzip)
2024-04-10T23:26:54.850091Z	  ├─ dist/_worker.js/chunks/runtime.mjs (204 kB) (46.2 kB gzip)
2024-04-10T23:26:54.850315Z	  ├─ dist/_worker.js/chunks/runtime.mjs.map (4.44 kB) (826 B gzip)
2024-04-10T23:26:54.850501Z	  └─ dist/_worker.js/index.js (144 B) (148 B gzip)
2024-04-10T23:26:54.850652Z	Σ Total size: 210 kB (47.8 kB gzip)
2024-04-10T23:26:54.854427Z	[success] [nitro] You can preview this build using `npx wrangler pages dev dist/`
2024-04-10T23:26:54.854736Z	[success] [nitro] You can deploy this build using `npx wrangler pages deploy dist/`
2024-04-10T23:26:54.895632Z	Finished
2024-04-10T23:26:54.896085Z	Found _worker.js in output directory. Uploading.
2024-04-10T23:26:55.760071Z	Attaching additional modules:
2024-04-10T23:26:55.766192Z	┌─────────────────────────┬──────┬────────────┐
2024-04-10T23:26:55.766466Z	│ Name                    │ Type │ Size       │
2024-04-10T23:26:55.766566Z	├─────────────────────────┼──────┼────────────┤
2024-04-10T23:26:55.766656Z	│ chunks/routes/index.mjs │ esm  │ 0.73 KiB   │
2024-04-10T23:26:55.766783Z	├─────────────────────────┼──────┼────────────┤
2024-04-10T23:26:55.766916Z	│ chunks/runtime.mjs      │ esm  │ 199.15 KiB │
2024-04-10T23:26:55.767054Z	└─────────────────────────┴──────┴────────────┘
2024-04-10T23:26:55.811841Z	✨ Compiled Worker successfully
2024-04-10T23:26:55.881448Z	Found _routes.json in output directory. Uploading.
2024-04-10T23:26:55.893425Z	Validating asset output directory
2024-04-10T23:26:56.574643Z	Deploying your site to Cloudflare's global network...
2024-04-10T23:27:00.7183Z	Uploading... (0/1)
2024-04-10T23:27:01.670487Z	Uploading... (1/1)
2024-04-10T23:27:01.670967Z	✨ Success! Uploaded 1 files (1.41 sec)
2024-04-10T23:27:01.671156Z	
2024-04-10T23:27:02.030702Z	✨ Upload complete!
2024-04-10T23:27:04.14264Z	Success: Assets published!
2024-04-10T23:27:06.252786Z	Success: Your site was deployed!

penalosa · 2024-04-10T23:35:11Z

@pi0 Because the wrangler.toml file is generated during the build, build-time environment variables won't be injected (which is what that initial error message refers to). However, bindings, compatibility flags, and runtime environment variables should all work for the deployed project

pi0 · 2024-04-11T01:21:17Z

@penalosa After some trial and error (and some pointers re wrangler validation thanks to @atinux) I found with a default name (any value is that ok?) and pages_build_output_dir (tested here) it only works with after build.

But it has the issue that if a framework wants to auto-enable a certain flag (like nodejs_als) it makes the bindings defined by user in UI, not work anymore and disabled because as I could understand, the auto-generated wrangler.toml becomes the only new source of trust. It makes it essentially a breaking change for our users when they configure bindings through UI.

Would _worker.bundle and _build.metadata.json in #5580 allow to support both user config and framework auto generated config combined?

(btw thank you so much for the quick response ❤️)

IgorMinar · 2024-04-11T05:06:16Z

@pi0 oof, I was hoping that we could avoid this because this will result in weird "source of truth" issues. This is why we design the wrangler.toml support in Pages to override and not augment the dash config.

Now I understand some of your comments. It seems that what you really want is an ability to intercept the runtime config creation in wrangler and augment it. Is that right?

penalosa · 2024-04-11T12:11:53Z

@pi0 we intentionally lock the dashboard in cases like this (where we would have to merge config from a file and from the UI), because it's not always clear how that merging should work. For instance, what if a user added some compatibility flags via the dashboard? In the Nuxt case ideally that list would be merged with the Nuxt provided list so that nodejs_compat was preserved, but that's not the only way to do that merging (potentially in another scenario the user would want UI flags to override config flags entirely). Because of that potential for conflict, when Pages config is managed via a config file we lock the dashboard UI to prevent confusing states.

However, what the linked PR would give you is a way to manage the merging of user config (from a file) and framework config yourself, within the framework build command. This would let frameworks individually decide on the specific logic they want to use for merging, although it wouldn't let users edit things through the dashboard.

IgorMinar added the enhancement New feature or request label Apr 10, 2024

workers-devprod added this to workers-sdk Apr 10, 2024

github-project-automation bot moved this to Untriaged in workers-sdk Apr 10, 2024

pi0 mentioned this issue Apr 10, 2024

refactor(cloudflare-pages): update root wrangler.toml in CI nitrojs/nitro#2355

Merged

penalosa mentioned this issue Apr 10, 2024

Allow custom _worker.bundle and _build-metadata.json files for advanced framework use cases #5580

Closed

9 tasks

petebacondarwin moved this from Untriaged to In Progress in workers-sdk Aug 12, 2024

petebacondarwin linked a pull request Dec 4, 2024 that will close this issue

feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚀 Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

🚀 Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

IgorMinar commented Apr 10, 2024 •

edited

Loading

penalosa commented Apr 10, 2024 •

edited

Loading

pi0 commented Apr 10, 2024

IgorMinar commented Apr 10, 2024

IgorMinar commented Apr 10, 2024

pi0 commented Apr 10, 2024 •

edited

Loading

penalosa commented Apr 10, 2024 •

edited

Loading

pi0 commented Apr 11, 2024

IgorMinar commented Apr 11, 2024

penalosa commented Apr 11, 2024 •

edited

Loading

🚀 Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

🚀 Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

Comments

IgorMinar commented Apr 10, 2024 • edited Loading

Describe the solution

penalosa commented Apr 10, 2024 • edited Loading

pi0 commented Apr 10, 2024

IgorMinar commented Apr 10, 2024

IgorMinar commented Apr 10, 2024

pi0 commented Apr 10, 2024 • edited Loading

penalosa commented Apr 10, 2024 • edited Loading

pi0 commented Apr 11, 2024

IgorMinar commented Apr 11, 2024

penalosa commented Apr 11, 2024 • edited Loading

IgorMinar commented Apr 10, 2024 •

edited

Loading

penalosa commented Apr 10, 2024 •

edited

Loading

pi0 commented Apr 10, 2024 •

edited

Loading

penalosa commented Apr 10, 2024 •

edited

Loading

penalosa commented Apr 11, 2024 •

edited

Loading