Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestions on how to run this within a Lambda environment #478

Open
tavosansal opened this issue Sep 10, 2024 · 3 comments
Open

Suggestions on how to run this within a Lambda environment #478

tavosansal opened this issue Sep 10, 2024 · 3 comments
Labels
question Further information is requested

Comments

@tavosansal
Copy link

Hello,
Thank you for this package. Really cool stuff.
Im running in an issue and wondering if anyone else has encountered it.

I am trying to run webR in a lambda environment with esbuild. I am running into the following error:

"Cannot find module '/var/task/webr-worker.js'","stackTrace":["Error: Cannot find module '/var/task/webr-worker.js'","    
at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)","    
at Module._load (node:internal/modules/cjs/loader:1051:27)","    
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12)","    
at MessagePort.<anonymous> (node:internal/main/worker_thread:186:26)","    
at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)","    
at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)"],"code":"MODULE_NOT_FOUND","requireStack":[]}}

I am bundling my lambda with esbuild using CDK with the default esbuild options for a NodejsFunction. I managed to track this down to this line (although there are other similar ones for different workers).

I took a look at my file after esbuild does the bundling and it looks like only the main file from the package is getting bundled:

// ../node_modules/webr/dist/webr.cjs
var require_webr = __commonJS({
  "../node_modules/webr/dist/webr.cjs"(exports2, module2) {

There are no other entries in the file about webr definitions.

Wondering if anyone has run into this and has any ideas on how to proceed? Appreciate the time.

@tavosansal
Copy link
Author

Following up. I was able to create a Lambda layer with the node_modules folder that includes WebR and that worked. So I am thinking it would be something with the configuration of the package that esbuild does not pick up the right parts? I am by no means a package export configuration expert so I will decline to comment further.

@georgestagg georgestagg added the question Further information is requested label Sep 11, 2024
@georgestagg
Copy link
Member

I don't know much about executing JavaScript in AWS lambda, though I will say generally that it's unlikely that with our current setup we can bundle everything into a single JavaScript file, because we rely on starting a Web Worker to execute R Wasm computation and the worker has it's own script: webr-worker.js.

It looks like this is the file that's getting missed by the bundler, your esbuild script might need to be configured to copy web-worker.js from the webR distribution (in node_modules/webr, also available as downloads on the Releases page of the GitHub repo) to the output directory.

It's not impossible to embed everything into a single file, but would require some setup to tell esbuild to embed the contents of webr-worker.js into webr.cjs/webr.mjs. There are also lots of additional support files that webR relies on finding to startup, including non-JavaScript filesystem images that are required so that R thinks it's running under a standard Unix filesystem. Again, I think embedding those files into a single output would be tricky, but perhaps esbuild can be told to copy them from node_modules/webr/vfs to a reasonable directory as part of bundler output.

I will leave the issue open in case others have any further advice or suggestions to add.

@tavosansal
Copy link
Author

Thanks for the reply.
I think the layer approach is ok for what I am trying to do. Technically you can expose the files the webr package needs via esbuild config for lambda. But I found it cumbersome to work with (I am using AWS CDK). So for now I am going to work with the layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants