Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mounting WORKERFS images from the main thread using the JS API #488

Merged
merged 2 commits into from
Sep 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@

* When explicitly creating a list using the `RList` constructor, nested JavaScript objects at a deeper level are also converted into R list objects. This does not affect the generic `RObject` constructor, as the default is for JavaScript objects to map to R `data.frame` objects using the `RDataFrame` constructor.

## Bug Fixes

* (Regression) Mounting filesystem images using the `WORKERFS` filesystem type now works again in the browser using the JavaScript API from the main thread (#488).

# webR 0.4.2

## New features
Expand Down
52 changes: 28 additions & 24 deletions src/webR/mount.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,31 @@ type WorkerFileSystemType = Emscripten.FileSystemType & {
contents: ArrayBufferView, mtime?: Date) => FS.FSNode;
};

/**
* Hooked FS.mount() for using WORKERFS under Node.js or with `Blob` objects
* replaced with Uint8Array over the communication channel.
* @internal
*/
export function mountFS(type: Emscripten.FileSystemType, opts: FSMountOptions, mountpoint: string) {
// For non-WORKERFS filesystem types, just call the original FS.mount()
if (type !== Module.FS.filesystems.WORKERFS) {
return Module.FS._mount(type, opts, mountpoint) as void;
}

// Otherwise, handle `packages` using our own internal mountImageData()
if ('packages' in opts && opts.packages) {
opts.packages.forEach((pkg) => {
mountImageData(pkg.blob as ArrayBufferLike, pkg.metadata, mountpoint);
});
} else {
// TODO: Handle `blobs` and `files` keys.
throw new Error(
"Can't mount data. You must use the `packages` key when mounting with `WORKERFS` in webR."
);
}
}


/**
* Download an Emscripten FS image and mount to the VFS
* @internal
Expand Down Expand Up @@ -88,29 +113,6 @@ export function mountImagePath(path: string, mountpoint: string) {
}
}

/**
* An implementation of FS.mount() for WORKERFS under Node.js
* @internal
*/
export function mountFSNode(type: Emscripten.FileSystemType, opts: FSMountOptions, mountpoint: string) {
if (!IN_NODE || type !== Module.FS.filesystems.WORKERFS) {
return Module.FS._mount(type, opts, mountpoint) as void;
}

if ('packages' in opts && opts.packages) {
opts.packages.forEach((pkg) => {
// Main thread communication casts `Blob` to Uint8Array
// FIXME: Use a replacer + reviver to handle `Blob`s
mountImageData(pkg.blob as ArrayBufferLike, pkg.metadata, mountpoint);
});
} else {
throw new Error(
"Can't mount data under Node. " +
"Mounting with `WORKERFS` under Node must use the `packages` key."
);
}
}

// Mount the filesystem image `data` and `metadata` to the VFS at `mountpoint`
function mountImageData(data: ArrayBufferLike | Buffer, metadata: FSMetaData, mountpoint: string) {
if (IN_NODE) {
Expand Down Expand Up @@ -140,7 +142,9 @@ function mountImageData(data: ArrayBufferLike | Buffer, metadata: FSMetaData, mo
WORKERFS.createNode(dirNode, file, WORKERFS.FILE_MODE, 0, contents);
});
} else {
Module.FS.mount(Module.FS.filesystems.WORKERFS, {
// Main thread communication casts `Blob` to Uint8Array
// FIXME: Use a replacer + reviver to handle `Blob`s
Module.FS._mount(Module.FS.filesystems.WORKERFS, {
packages: [{
blob: new Blob([data]),
metadata,
Expand Down
8 changes: 5 additions & 3 deletions src/webR/webr-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { WebRPayloadRaw, WebRPayloadPtr, WebRPayloadWorker, isWebRPayloadPtr } f
import { RPtr, RType, RCtor, WebRData, WebRDataRaw } from './robj';
import { protect, protectInc, unprotect, parseEvalBare, UnwindProtectException, safeEval } from './utils-r';
import { generateUUID } from './chan/task-common';
import { mountFSNode, mountImageUrl, mountImagePath } from './mount';
import { mountFS, mountImageUrl, mountImagePath } from './mount';
import type { parentPort } from 'worker_threads';

import {
Expand Down Expand Up @@ -840,8 +840,6 @@ function init(config: Required<WebROptions>) {

Module.preRun.push(() => {
if (IN_NODE) {
Module.FS._mount = Module.FS.mount;
Module.FS.mount = mountFSNode;
globalThis.FS = Module.FS;
(globalThis as any).chan = chan;
}
Expand All @@ -852,6 +850,10 @@ function init(config: Required<WebROptions>) {
Module.ENV.HOME = _config.homedir;
Module.FS.chdir(_config.homedir);
Module.ENV = Object.assign(Module.ENV, env);

// Hook Emscripten's FS.mount() to handle ArrayBuffer data from the channel
Module.FS._mount = Module.FS.mount;
Module.FS.mount = mountFS;
});

chan?.setDispatchHandler(dispatch);
Expand Down