Missing WebIDL processing features overview

[MOZGIII]

I was doing a manual implementation for the WebTransport types, and here's case-study-based a list of missing features that makes WebIDL codegen really unusable:

1. Lack of getters support for dictionaries. A lot of dictionaries are used in return positions: ReadableStream, pretty much everywhere in the getStats for WebTransport, etc.
2. Dictionaries setters do not allow unsetting values (for the most part) - we might want to either make them all take Option<T> instead of T, or generate an additional accessor to allow unset_....
3. Untyped async calls - the WebIDL has enough data to properly deduce the type, but that is not done, and an untype js_sys::Promise it returned instead. We should consider adding types support to the js_sys::Promise, and also generating the calls that return promises as real async fns. This is, however, blocked by Add .into() to the generated code for async fns #3919, as async fn generation is broken.
4. Well, Add .into() to the generated code for async fns #3919. The async fn generation is broken.
5. The catch is not generated correctly. This is an issue with WebIDL definitions, as they lack the machine-readable data on whether a call can throw or not - but nonetheless I count this towards the list of issues of the WebIDL-generated rust code.
6. ReadableStream/WriteableStream should have a generic indicating what type they produce - they are capable of streaming not just the bytes (i.e. binary data, where they act like Read/Write), but objects too (acting more like a stream / async iterator in rust sense).
7. Inability to turn certain hot calls final at the codegen level.
8. I can't just install the latest wasm_bindgen_webidl as a dependency and use it in my crate - as the latest version is not published, and the last published is very old and conflicts with the rest of the crates. This is by far the easiest issue to solve.
9. Errors types are not properly indicated anywhere, while the WebIDL has relevant info.

Also, some nits I encountered while implementing things manually.

1. missing_docs lint is triggered for wasm_bindgen macro enums.
2. No explicit support for attributes as wasm_bindgen macro, as in syntax sugar for generating both getter and setter from the same attribute.
3. Declaring dictionaries manually is too verbose, they should really be a first-class item in the macro.

Not sure this is the right place for this, and those issue are most definitely reported before - but this is a case study organized in one place, and will hopefully act like a concise roadmap towards a better support for WebIDL. Hopefully if all those are solved somehow we'd get a way nicer codegen.
For now, using WebIDL is much more painful than using custom type declarations.

[pablosichert]

Dictionaries setters do not allow unsetting values (for the most part) - we might want to either make them all take Option<T> instead of T, or generate an additional accessor to allow unset_....

Just a quick comment on this one: allowing to unset values would effectively void the conditions that should hold for reading the properties. Meaning that all getters would need to be wrapped in an Option.

I guess you could implement the unset_... functions anyway and mark them unsafe?

[MOZGIII]

Not all of them - the attribute ? (with ? at the end) - ones only, those should be unsettable, as they can effectively be not set and read as null (as I understand - null specifically, not undefined). For those values we absolutely need getters to return Option.

UPD: oh, yeah, dictionaries, right. They don't have definitions like that, and thus must absolutely always do the explicit Option<T> for all values. In my opinion, it is not about the ergonomics - but about correctness. In the context of a dictionaries, WebIDL does not actually encode the expected default values - so we can't procedurally generate them - but the spec itself usually does, so it is possible for a developer to properly deduce what value will be in effect if a dictionary value is unset, on a case-by-case basis via reading the docs. Combined with limitations of panics at wasm, and this becomes the only solution to avoid super-fragile apps imo.

[pablosichert]

In the context of a dictionaries, WebIDL does not actually encode the expected default values

They actually do, so I think your suggestion of allowing to unset optional values only would work great.

E.g. the first dictionary declaration I could find going alphabetically in the enabled WebIDLs:

wasm-bindgen/crates/web-sys/webidls/enabled/AnalyserNode.webidl

Lines 13 to 18 in e78db23

	dictionary AnalyserOptions : AudioNodeOptions {
	unsigned long fftSize = 2048;
	double maxDecibels = -30;
	double minDecibels = -100;
	double smoothingTimeConstant = 0.8;
	};

It just looks like the code generation for dictionary constructors produces undefined behavior by not setting default values in new:

wasm-bindgen/crates/web-sys/src/features/gen_AnalyserOptions.rs

Lines 30 to 38 in e78db23

	impl AnalyserOptions {
	#[doc = "Construct a new `AnalyserOptions`."]
	#[doc = ""]
	#[doc = "*This API requires the following crate features to be activated: `AnalyserOptions`*"]
	pub fn new() -> Self {
	#[allow(unused_mut)]
	let mut ret: Self = ::wasm_bindgen::JsCast::unchecked_into(::js_sys::Object::new());
	ret
	}

I'll open a separate issue for that.

[daxpedda]

1. Lack of getters support for dictionaries. A lot of dictionaries are used in return positions: ReadableStream, pretty much everywhere in the getStats for WebTransport, etc.

Covered by #1793.

2. Dictionaries setters do not allow unsetting values (for the most part) - we might want to either make them all take Option<T> instead of T, or generate an additional accessor to allow unset_....

I don't think this is something we want to add generally to all dictionary types without some well justified use-case.
But I think it would be a nice feature to be able to generate a binding for unsetting properties.

3. Untyped async calls - the WebIDL has enough data to properly deduce the type, but that is not done, and an untype js_sys::Promise it returned instead. We should consider adding types support to the js_sys::Promise, and also generating the calls that return promises as real async fns. This is, however, blocked by Add .into() to the generated code for async fns #3919, as async fn generation is broken.

I think introducing a typed Promise would be an interesting idea to explore, though I'm unsure whats preventing higher-level libraries like gloo to implement this instead.

However having the WebIDL generator use async fn() -> T instead of returning Promise would be a mistake I believe, as this introduces overhead that is not always necessary. E.g. sometimes you just want to do something with the Promise type itself or pass it to something else without having the result go through Wasm.

4. Well, Add .into() to the generated code for async fns #3919. The async fn generation is broken.

Reviewing this is in my backlog.

5. The catch is not generated correctly. This is an issue with WebIDL definitions, as they lack the machine-readable data on whether a call can throw or not - but nonetheless I count this towards the list of issues of the WebIDL-generated rust code.

AFAIK there isn't anything wasm-bindgen can do about this apart from manually adding catch when necessary, right?

6. ReadableStream/WriteableStream should have a generic indicating what type they produce - they are capable of streaming not just the bytes (i.e. binary data, where they act like Read/Write), but objects too (acting more like a stream / async iterator in rust sense).

Similar to Promise.

7. Inability to turn certain hot calls final at the codegen level.

Unfortunately I don't know what that means.

8. I can't just install the latest wasm_bindgen_webidl as a dependency and use it in my crate - as the latest version is not published, and the last published is very old and conflicts with the rest of the crates. This is by far the easiest issue to solve.

I wasn't aware that anybody would use this outside of wasm-bindgen, but I'm happy to review a PR including this in our release process unless a good reason comes up not to.

9. Errors types are not properly indicated anywhere, while the WebIDL has relevant info.

Really? I didn't encounter something like this yet I believe, but yeah, I wouldn't know why not to take that into account.
Though this might be a breaking change that we can't introduce now.

1. missing_docs lint is triggered for wasm_bindgen macro enums.

A quick try showed that attributes aren't passed on to the variants, so yeah, that should be fixed.

2. No explicit support for attributes as wasm_bindgen macro, as in syntax sugar for generating both getter and setter from the same attribute.

I'm not sure how much convenience I want to add directly to wasm-bindgen in this department.
But I'm open to discussing it.

3. Declaring dictionaries manually is too verbose, they should really be a first-class item in the macro.

Hm, again, not too sure what to think of this. Open to discussing more specific improvements.

Not sure this is the right place for this, and those issue are most definitely reported before - but this is a case study organized in one place, and will hopefully act like a concise roadmap towards a better support for WebIDL. Hopefully if all those are solved somehow we'd get a way nicer codegen. For now, using WebIDL is much more painful than using custom type declarations.

The WebIDL generator is not meant for public consumption and I don't believe this is something we want to take on with the limited maintainer resources we have, considering this can all be done externally. Though obviously any improvements to web-sys is something that we are definitely interested in.

---

I appreciate the input, its good to hear from users to see where exactly we can improve things!
I'm happy to discuss details in follow-up issues or PRs!

[MOZGIII]

5. Untyped async calls - the WebIDL has enough data to properly deduce the type, but that is not done, and an untype js_sys::Promise it returned instead. We should consider adding types support to the js_sys::Promise, and also generating the calls that return promises as real async fns. This is, however, blocked by Add .into() to the generated code for async fns #3919, as async fn generation is broken.

I think introducing a typed Promise would be an interesting idea to explore, though I'm unsure whats preventing higher-level libraries like gloo to implement this instead.

However having the WebIDL generator use async fn() -> T instead of returning Promise would be a mistake I believe, as this introduces overhead that is not always necessary. E.g. sometimes you just want to do something with the Promise type itself or pass it to something else without having the result go through Wasm.

Ideally we'd want the Promise to implement IntoFeature, and also be typed. That way we'd have the best of both world: the ergonomics of await-able functions and the control of having proper Promise objects.

I suggest altering the Promise to be Promise<T = Unknown> - i.e. promise with an option to add a type and a fallback to an unknown type (type Unknown = wasm_bindgen::JsValue;, or use wasm_bindgen::JsValue directly).

[MOZGIII]

17. Errors types are not properly indicated anywhere, while the WebIDL has relevant info.

Really? I didn't encounter something like this yet I believe, but yeah, I wouldn't know why not to take that into account. Though this might be a breaking change that we can't introduce now.

See my hand-crafted code for WebTransport for an example: MOZGIII/xwt#152
The error information is present at WebIDL. It is a bit messy, and would require some extra parsing to pull out from the files. Doing it manually was a good exercise.

[daxpedda]

I suggest altering the Promise to be Promise<T = Unknown> - i.e. promise with an option to add a type and a fallback to an unknown type (type Unknown = wasm_bindgen::JsValue;, or use wasm_bindgen::JsValue directly).

It would be Promise<T = Result<JsValue, JsValue>>.
Yes, I agree this approach would solve the issue with many untyped APIs we currently have and is also what I would go for as well.
Unfortunately we wouldn't be able to make use of such a system because it would require a breaking change in web-sys, so this is where we are stuck at.