Rust design question: nicely abstracting over WASMI and wasmtime

[dominic-mulligan-arm]

Chihuahua has a fairly horrible kludge hidden within it, and the recent wasi-host-abi changes have me re-addressing all of this code, so I am wondering whether anybody can see a decent way of removing it.

In short, the problem is caused by how WASMI and wasmtime handle the routing of host functions in the respective execution engines. WASMI uses a system centred around traits, where a type capturing the runtime state of the host is created and registered as an instance of the appropriate routing traits, which dispatch on the name of the host function being called, call the appropriate function on the underlying runtime state struct, and then feed the result back to the executing WASM program.

You can see the current Chihuahua implementation of that, here, where ModuleImportResolver is the WASMI trait that does host-call dispatch, Externals in the trait implementation immediately below is the trait that actually implements host-calls by calling methods on this runtime state and routes the result back, and WasmiHostProvisioningState is the runtime state of the Veracruz runtime.

On the other hand, wasmtime takes a completely different approach, and doesn't use traits. In particular, when invoking a program with wasmtime you must iterate through all of the program's imports, inspect their name, and if appropriate register what is essentially a closure into the wasmtime runtime that implements the host-call in question.

You can see the current Chihuahua implementation of this here.

Note that, as a result of this, and our need to keep a runtime state of data sources, program, and so on around, we still need a struct that captures this runtime state for the wasmtime implementation. At the moment, this is registered as a lazy_static global, protected by a Mutex, which the closures implementing each host-call read and modify as appropriate. You can see in e.g. the implementation of the getrandom host-call, here, for instance that we take a lock on this object in order to implement the call.

This poses a problem, as Chihuahua needs to provide a uniform interface to the rest of the Veracruz runtime irrespective of what the underlying execution engine actually is. At present, this interface is represented as a trait, Chihuahua, defined here. Using this trait, we can create a "factory method" that returns "some" implementation of the trait, as seen here. Then, the rest of the Veracruz codebase can obtain a trait object backed by the appropriate execution engine, either WASMI or wasmtime, simply by passing a flag to that factory function which dispatches as appropriate.

The problem is that implementing this Chihuahua trait is easy for WASMI (we have a WASMIHostProvisioningState which this trait can be implemented on, and as WASMI also implements all of its routing traits on this type too, everything lines up) but becomes horrible for wasmtime.

You can see the problem here.

Note that I need to introduce a new, dummy type to implement the Chihuahua trait on. Then, every method in the trait completely ignores the &self parameter (which is now bound to a reference to an instance of the dummy type) in favour of taking a lock on the "real" runtime state protected by a Mutex in the lazy_static. This is pretty horrible from a design point-of-view, and also probably from a performance point of view where a Mutex needs to be acquired on every call.

So: can anybody see an alternative way of providing a uniform interface for Chihuahua that doesn't involve a horrible hack like this?

[geky]

Does this problem come from invoke_entry_point being a global standalone function?

veracruz/chihuahua/src/hcall/wasmtime.rs

Line 333 in 5090a11

pub(crate) fn invoke_entry_point() -> Result<i32, Trap> {

Would it go away if you only allow invoke_entry_point to be called as a member function?

veracruz/chihuahua/src/hcall/wasmi.rs

Line 851 in 5090a11

pub(crate) fn invoke_entry_point(&mut self) -> Result<i32, FatalHostError> {

If you did that I think you can then enforce the appropriate lifetimes between the Wasmtime Module/Instance structs and the callbacks + Chihuahua struct.

[ShaleXIONG]

In term of working with Rust and the design, a further challenge I have in #78 is object lifetime.

For some background, in #78, I have already reduced the chihuahua interface to file system API plus an extra launch interface for running a program in FS (recall that we might have several programs in the file system). However, the FS has a longer lifetime than the executing engine, which means some or all file system API are callable without an existing engine, e.g. checking if a file exists, read or write to a file. Note that data providers, program providers and result retrievers rely on these FS API for interacting with Veracruz. However, the launch API (a combination of old load_program and invoke_entry_point) requires the existence of an engine and a VFS and I am working on instantiating execution engine on-demand rather than keeping them in memory all the time.

I am currently wrapping FS data in a lazy_static, and the implementation of FS API in Chihuahua becomes wrappers on the global FS data. In an ideal situation, we should decouple the FS API and execution engine API, and the FS reference/handler is passed around between execution engine instances. No idea how to do so as Rust has a very strict ownership type system. Possibly Rc will do.

[nspin]

I think that we should admit a third sort of implementation, in addition to those usingwasmi and wasmtime as libraries, into our discussion of a better abstraction. Some isolation backends, such as Nitro enclaves and seL4, allow for process-level sandboxing within an isolate. On such platforms, we can run wasmtime, both the JIT compilation and execution of the JIT-ed code, in a process-level sandbox. This has already been implemented for the seL4 backend.

In this sort of implementation, some SDK calls can be handled entirely within the process-level sandbox, whereas others, such as those involving computation input and output, will require IPC between the top-level Veracruz runtime and the process-level sandbox it manages.

[nspin]

This is pretty horrible from a design point-of-view, and also probably from a performance point of view where a Mutex needs to be acquired on every call.

I think it it reasonable to expect that, for all of the isolation backends we support, the implementation of Mutex imposes minimal overhead (just a few atomic instructions) in the absence of contention.