How Should We Test wgpu

[cwfitzgerald]

Premise

Now that #1538, which introduced our system of core tests, has merged and some residual issues with it have been solved. It's time to start thinking about what tests actually need to be written. I want to put together a proposal/list of the kinds of tests that need to be written, but I wanted to open this up to the community before I write up a formal list which we can issue a call to action for.

Defense in Depth

i.e. where do these tests fit in the world?

These core tests aren't the only tests that will be checking the validity of our code. The WebGPU CTS (conformance test suite) is a standardized test suite for all WebGPU implementations. This tests a lot of the more mechanical and validation-based operations. Through #1421, we should be able to run CTS in CI at some point through our friends at deno, so we can consider it a part of our active defenses.

The example reftests, part of our core tests, help us by making sure our end-to-end process works well enough for the scenes in the examples. While the examples do cover quite a few features, they do not test them individually or thoroughly.

Distributed Testing

As a future goal of #1379, we will build a distributed testing network such that we can automatically run all of our core tests across a large array of hardware. This network will likely not test the WebGPU CTS due to the required infrastructure to check it.

My Current Ideas

The following is my current idea of what we should or shouldn't test in these core tests. These are categories and would not represent individual tests.

Should Test

• CTS-like tests for native features (these won't get covered by the normal WebGPU CTS)
• Backend specific issues (ex. do we properly compensate for D3D and GL using gl_VertexID vs gl_VertexIndex)
• Device specific issues (ex. when we implement a work around for a driver bug, does that bug stay gone)
• How we handle in exceptional situations (ex. are our limits accurate enough such that we can always use every last thing without issues)
• Feature tests which validate one particular feature is working correctly, especially across backends. [meta] Feature Testing Checklist #807 has a starting list of possible feature tests.

Shouldn't Test

• Specific algorithms within wgpu-core. This should happen in standard unit tests.
• CTS-llke tests for core WebGPU features,

Questions

• Do you disagree with any point on this list?
• Are there any kinds of tests you think should/shouldn't be tested that aren't listed?
• Are there any particular specific tests that we should make sure to include?

[kvark]

This sounds very reasonable. It's also not a point of bifurcation: we can start armed with your list and then adjust as we go.

One such adjustment could be to upstream several categories of these tests into the main CTS. This would actually be everything in "Should test" except for "CTS-like tests for native features". I think we should make the best effort to at least try upstreaming, and ask our partners to do the same. In this case, our cumulative benefit would be the greatest :)

[cwfitzgerald]

It's also not a point of bifurcation: we can start armed with your list and then adjust as we go.

💯. The main point of this is to see if there is anything obvious that I had missed.

This would actually be everything in "Should test" except for "CTS-like tests for native features".

I definitely agree this is the longer term solution as the more robust the WebGPU CTS is the better.

It does get to a couple of my worries about dealing with the WebGPU CTS in our testing which revolve around the necessity of using deno as a runtime. These are all solvable, but possibly not simply.

a) distributed testing of the CTS becomes difficult because we don't control the testing infrastructure, so hooking into it in a way to facilitate and phone home the results of the test will be difficult. Similarly hooking up to a testing rig might involve changes to deno.
b) if we start to rely on the CTS over deno for testing, it means that any breaking change in wgpu-core will require a change in deno before we can continue testing. Same goes for bugs in deno (even if rare).

This definitely isn't an argument against upstreaming the tests, we just need to make sure we retain the strength and comprehensiveness that our testing infrastructure is shaping up to be.