Origin Private File System

[rhashimoto]

I created an example VFS for the proposed Origin Private File System API using the dependent proposal for AccessHandle. There is a benchmark page here - note that this currently works only on Safari 15.2+ running on macOS 12.2+, Chrome 102+, and Edge 102+.

My early results have the benchmarks running generally slower than IndexedDB (a lot slower for Test 1), testing mainly on Chrome Canary. Part of the reason is it looks like Chrome has made some significant performance improvements with IndexedDB because it seems a lot faster than the last time I was playing with benchmarks. Another factor is the extra work put into the IndexedDB VFS. I haven't yet considered whether these or other optimizations can be used with OPFS.

I do know that programming the OPFS VFS is a much more pleasant experience than IDB (except for the fact that AccessHandle requires a Worker). The code can be simpler and less prone to lurking bugs. Once OPFS is stable and widely supported, I think it might be the best persistence choice when performance is not the absolute priority.

[rhashimoto]

OPFS has reached the stable versions of Safari, Chrome, and Edge. The tracking bug for Firefox is here (in progress but looks a ways off). The OPFS-specific benchmark page won't work on Firefox even if OPFS support were complete because that page uses ES6 modules in a worker which is also not yet supported in Firefox (and because I'm too lazy to bundle the worker code). Hopefully both features will arrive soon.

[sgbeal]

Good early morning, Roy!

i'm currently working on the sqlite project's WASM-related documentation (to be released soon along with v3.40) and i have been citing your work as the first known implementation of wasmified sqlite using OPFS. Insofar as you know, is that correct?

[sgbeal]

I think I read that the folks proposing OPFS at Google had an implementation of SQLite running,

i have it on very good authority that the OPFS folks approached the sqlite team about implementing that earlier this year. We (the sqlite team, where i'm the newly-crowned "JS/WASM Guy") initially used Emscripten's WASMFS for OPFS storage, but that was a stopgap measure for us while we developed an sqlite3_vfs for OPFS, which we didn't have running until sometime in September, shortly after we saw your VFS impl.

If you're only counting publicly available implementations, then wa-sqlite is the first working one I know of.

Yeah, if it wasn't published publicly it doesn't count ;). You're the winner, as far as i'm concerned :).

Is the webkitFileSystem a predecessor of OPFS or is it what OPFS used to be called?

More info about the sqlite project's wasm effort, for those who enjoy that kind of thing, can be found at https://sqlite.org/wasm. If you'll check the bottom of the "about" page you'll find yourself immortalized there :).

[rhashimoto]

Cool!

Just one question with this bit:

As of late 2022, only bleeding-edge versions of Chrome and Chromium have the necessary APIs.

Is this really true? I'd be especially surprised if it didn't work with Edge which doesn't lag that far behind Chromium. And while I have grumbled a lot about missing APIs in Safari, I have been much happier with it recently.

[rhashimoto]

Oh, yes, as I understand the history, OPFS is an extension to the File System Access API. But there were strong objections to the full FS API so it's just OPFS that seems to be moving forward now.

[sgbeal]

Is this really true? I'd be especially surprised if it didn't work with Edge which doesn't lag that far behind Chromium.

You lost me at "Edge" ;). If it doesn't run on Linux then i have no clue :).

That said, i'll rephrase those docs to refer to Chromium-derived browsers as a whole.

And while I have grumbled a lot about missing APIs in Safari, I have been much happier with it recently.

You lost me at "Safari" ;).

All development and testing of the wasm/js stuff has been done using Firefox and Chrome/Chromium on Linux, but FF doesn't yet do OPFS (it will once they see all new webapps in 2023 targeting Chrome for sqlite3-plus-OPFS support ;)). i (thankfully) don't have access to any Windows machines to test on and my deadly allergy to Safari's owning company prohibits me from laying a finger on one of their devices. i can't provide anything more than moral support for that particular user group.

[steida]

@sgbeal You wrote: "We experimented with that approach but it's... well, absurd 😉. It's too slow for anything beyond trivial databases and it's easy to get a database in an inconsistent state by visiting the page from two tabs."

Performance is not as bad at all and only absurd-sql is buggy for two tabs. I'm using wa-sqlite and it works properly.

[sgbeal]

AFAIK, "absurd" only ever demonstrated sqlite-in-IndexedDB, not OPFS?

[steida]

@sgbeal You are right, sorry.