Biome JS runtime: What engine to use?

[arendjr]

Introduction

Since we want to add the ability to have JS plugins in Biome, we have to make a decision: Which engine or runtime to use. In order to make an informed decision, we should compare engines across various criteria.

Criteria

The following is a list of criteria that can be used for evaluating various engines. Different people will likely value different criteria, so if you feel some criteria should weigh more heavily than others, please let us know your motivation!

• Runtime performance. Seems obvious, everybody likes their scripts to be fast. And yet we should not overstate the importance of runtime performance. Plugins are expected to perform most code matching through native Rust code, and long-running scripts would likely be discouraged anyway.
• Startup performance. Biome commands often run in milliseconds, rather than seconds. Scripts should start quickly, or they might have a disproportionate impact on overall performance.
• Binary size. Biome's executable is about 20MB in size. If possible, it would be nice if we could avoid bloating it too much.
• Language. The language in which an engine or runtime is implemented isn't the most important thing. But having smooth Rust bindings would help Biome maintainers to be more productive with it, while engines that use complex build tooling might be a chore to maintain.
• Documentation. Similarly, the better an engine's documentation, the more likely we are to successfully use an engine. Poor documentation may drain motivation from maintainers.
• Security. Engines can be complex beasts, and they're a common target for exploits. And as soon as users are able to share Biome plugins, likely through NPM and other platforms, there's a potential for abuse. Ideally, we would have a well-isolated sandbox with little surface for vulnerabilities.

Unfortunately, I do not have the time to try out each engine with Biome, so I cannot make a fully scientific comparison based on these criteria. Instead, I will do my best to estimate each engine's ability along the values of Great, Good, Average, Mediocre, and Poor, based on their documentation and technology choices. If you feel I have misjudged an engine's ability somewhere, please let me know so I can correct it.

Non-criteria

Some concerns are explicitly not taken into account:

• Memory usage. It would be good to include memory usage as well. Unfortunately, without actual benchmarks, I feel my guesses would be too unreliable to make a fair comparison.
• TypeScript compatibility. We anticipate that Biome will be able to add TypeScript compatibility regardless of engine, by doing the transformation itself.
• NPM compatibility. This one may be controversional, but I don't think we should factor in NPM compatibility for the simple reason that Biome doesn't require NPM to run either. So it would be unfortunate if plugins rely on NPM compatibility, because that would de facto mean that Biome's ecosystem will still become tied to NPM. Instead, if plugins want to use NPM dependencies, they can bundle them before distribution.

Because compatibility with NPM (or other high-level runtimes) is excluded, we can also confine our focus on engines, rather than full runtimes. This also eliminates the worst offenders when it comes to binary size and startup performance. This eliminates Deno, Node.js, Bun, AWS LLTT, and workerd as options, but leaves the engines they are based on.

Contestants

With no further ado, these are engines we might consider:

V8

The engine used by Chrome and its siblings, Node.js, and Deno.

• Runtime performance: Great
• Startup performance: Mediocre
• Binary size: Average
• Language: C++ but good Rust bindings thank to Deno
• Documentation: Great -- https://v8.dev/docs
• Security: Mediocre -- a highly complex C++ project with a large surface area

JavaScriptCore

The engine used by Safari and Bun.

• Runtime performance: Great
• Startup performance: Good
• Binary size: Average
• Language: C++
• Documentation: Mediocre
• Security: Mediocre

SpiderMonkey

The engine used by Firefox and Servo.

• Runtime performance: Great
• Startup performance: Mediocre
• Binary size: Average
• Language: C++ but good Rust bindings thanks to Servo
• Documentation: Good -- https://spidermonkey.dev/docs/
• Security: Mediocre

QuickJS

QuickJS is a small and embeddable Javascript engine.

• Runtime performance: Average
• Startup performance: Great
• Binary size: Great
• Language: C but Rust bindings available. Requires patching to build on Windows.
• Documentation: Good -- https://docs.rs/rquickjs/latest/rquickjs/index.html
• Security: Average -- smaller surface area than the JIT engines

Boa

An ECMAScript engine written in Rust.

• Runtime performance: Mediocre
• Startup performance: Great
• Binary size: Great
• Language: Rust
• Documentation: Good -- https://docs.rs/boa_engine/latest/boa_engine/
• Security: Good -- Rust with very minor unsafe usage

Hermes

• Runtime performance: Average
• Startup performance: Great
• Binary size: Great
• Language: C++
• Documentation: Mediocre
• Security: Average

Note: Hermes offers various kinds of precompilation optimizations that are great for use cases such as React Native, but which might be inconvenient or troublesome for an environment such as Biome where scripts would be distributed in source form. As such, I expect its performance to be roughly on par with QuickJS, but not to surpass it.

Poll

There is a poll associated to gather a sense of community opinion. This poll is strictly non-binding and we reserve any right to make our own decision based on the tradeoffs we value. This is merely intended to get a sense of which choices would be preferred by the community, so we also encourage people to comment and explain their choice!

[arendjr]

As for my personal recommendation, I would advise to choose Boa:

• Being written in Rust, it is really easy to get started with and I was able to create a PoC in about an hour: Implement loading biome.config.js #1825
• Their APIs seem intuitive and appear well-designed. Documentation was good too.
• No weird shenanigans with custom build tools.
• They have great ECMAScript conformance. Of the engines discussed, only V8 and JavaScriptCore score better on https://test262.fyi.
• It has good to great rating on all the things we want except runtime performance.
• Runtime performance did improve considerably in the recent v0.18 and it seems they have plenty of room to improve.

[Conaclos]

I could add a “maturity” point. Boa and Hermes are not “stable” yet. However, Boa is working on their first stable release and the API seems mostly stable at this point. Regarding performance, it is not their current focus. We could expect improvements in the future.

Reading your comparison it seems that QuickJS is the best option. However, a single guy mostly maintains it. Boa seems a better choice on this regard. And because it is written in Rust, it can attract new contributors.

[arendjr]

Yeah, apart from the relative lack of maintenance, I would also consider the less-than-ideal Windows support for QuickJS kind of a dealbreaker. At least I don’t have a Windows machine, and I imagine few of us will be happy to deal with Windows-specific CI issues. Whereas for people that do use Windows, it also might be an extra hurdle to get things to build properly.

[jedel1043]

Regarding performance, it is not their current focus. We could expect improvements in the future.

Oh, definitely! We're reaching a point on the development of Boa where we are in a pretty good place on the conformance aspect (almost 90% passed tests on Test262), so we were planning to shift our main priority to performance. Expect some speedups on the engine sooner rather than later :)

[voxpelli]

Have you considered to instead go a WASM approach similar to what dprint has done?

Then you could leverage something like https://github.com/extism/js-pdk to create JS-based plugins (I believe its QuickJS behind the scenes) and use something like https://crates.io/crates/extism to run them.

A bonus then is that one can write the plugins in all kinds of other languages as well: Rust, Go, C#, F#, C, Haskell, Zig, AssemblyScript etc. And you get the encapsulation and security features that WebAssembly offers.

[arendjr]

Yes, it was part of our initial RFC: #1649

So the main reason to focus on JS/TS first is because the overwhelming majority of our users want that, and implementing JS plugins on top of WASM would make it much harder for us to reach that goal and also sacrifice performance in the process. We may still add WASM support at some point, but it's not our focus now.

[voxpelli]

One argument for going with QuickJS could be that it’s the engine that is most often used in WASM (Fastly use SpiderMonkey instead) and probably so because it eg lacks JIT

[conartist6]

Would it not be possible to use whatever JS runtime is loading the @biomejs/biome package?

It would be pretty odd for an npm package to ship its own JS runtime, and it's seems to me like there would be no way for your unpacked package size to stay as low as 200KB where it currently sits.

[conartist6]

Can QuickJS even run arbitrary code? Most of the projects mentioned here are interpreters, but QuickJS is a compiler.

[morgante]

QuickJS can certainly be used with arbitrary code. It powers the JS runtime in GritQL.

[voxpelli]

Source on that: https://bellard.org/quickjs/quickjs.html#Script-evaluation

Use JS_Eval() to evaluate a script or module source.

If the script or module was compiled to bytecode with qjsc, it can be evaluated by calling js_std_eval_binary(). The advantage is that no compilation is needed so it is faster and smaller because the compiler can be removed from the executable if no eval is required.

[conartist6]

I don't think Biome could use QuickJS unless it shipped the QuickJS compiler (far bigger than the hello world program) and then you'd still be compiling and running x86 binaries on the client system which is sketchy as regards security. So you'd have to just ask people to compile their plugins with the QuickJS compiler at which point you'd have OS-level plugins (and thus platform-specific plugins) rather than scriptability.

[conartist6]

Ah ok, sorry, JS, doesn't auto-refresh threads as you're typing. I see that now.

[voxpelli]

The Deno bindings for Rust is now considered stable: https://deno.com/blog/rusty-v8-stabilized