[Build] WASM build of v1.20.1 with --use_xnnpack fails

[raphaelmenges]

Describe the issue

Hello 👋

I want to utilize the ONNX runtime in a Web application. I successfully built the static library of ONNX runtime with SIMD and threading and linked it within my Emscripten project. The CPU execution providers works great. Now I tried to also include the XNNPACK execution provider with the --use_xnnpack flag to improve the performance of the inferencing.

However, the src/amalgam directory is missing from the XNNPACK distribution. xnnpack.cmake says:

# See source lists in _deps/googlexnnpack-src/BUILD.bazel for wasm_prod_microkernels

I cannot find any documentation at the XNNPACK how to manually generate these amalgam (?) microkernels. I also cannot find an example in this repository. How to build the XNNPACK execution provider for the Web?

Urgency

No response

Target platform

WASM

Build script

onnxruntime/build.sh \
    --build_dir ./onnxruntime-build \
    --config Release \
    --build_wasm_static_lib \
    --enable_wasm_simd \
    --enable_wasm_threads \
    --skip_tests \
    --disable_wasm_exception_catching \
    --disable_rtti \
    --use_xnnpack \
    --parallel

Error / output

CMake Error at external/xnnpack.cmake:189 (target_sources):
  Cannot find source file:

    ./onnxruntime-build/Release/_deps/googlexnnpack-src/src/amalgam/gen/scalar.c

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
  .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
  .f95 .f03 .hip .ispc
Call Stack (most recent call first):
  external/onnxruntime_external_deps.cmake:566 (include)
  CMakeLists.txt:614 (include)

Visual Studio Version

No response

GCC / Compiler Version

No response

[raphaelmenges]

Apparently there was an update of the XNNPACK dependency in b94ba09 from google/XNNPACK@0da379f to google/XNNPACK@309b75c. The old version of the XNNPACK dependency still contained the src/amalgam directory. As I understand it, this directory could be created once but there is no more such a script.

[raphaelmenges]

I could make the Release build configuration compile by removing the source files of src/amalgam from ./cmake/external/xnnpack.cmake:

  # kernels
+ # list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/scalar.c)
+ # list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/wasm.c)
- list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/scalar.c)
- list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/wasm.c)

  if(onnxruntime_ENABLE_WEBASSEMBLY_SIMD)
+   # list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/wasmsimd.c)
-   list(APPEND wasm_srcs ${XNNPACK_DIR}/src/amalgam/gen/wasmsimd.c)
    target_compile_options(XNNPACK PRIVATE "-msimd128")
  endif()

However, there are following errors when linking libonnxruntime.a:

[ 75%] Linking CXX executable example.js
em++: warning: -pthread + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 [-Wpthreads-mem-growth]
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: pthreadpool_create
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: xnn_deinitialize
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: pthreadpool_destroy
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: xnn_initialize
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: xnn_deinitialize
wasm-ld: error: libonnxruntime.a(xnnpack_execution_provider.cc.o): undefined symbol: pthreadpool_destroy
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_delete_weights_cache
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_create_average_pooling2d_nhwc_f32
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_create_average_pooling2d_nhwc_qu8
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_delete_operator
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_delete_operator
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_reshape_average_pooling2d_nhwc_f32
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_reshape_average_pooling2d_nhwc_qu8
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_setup_average_pooling2d_nhwc_f32
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_setup_average_pooling2d_nhwc_qu8
wasm-ld: error: libonnxruntime.a(average_pool.cc.o): undefined symbol: xnn_run_operator
wasm-ld: error: libonnxruntime.a(conv_base.cc.o): undefined symbol: xnn_create_deconvolution2d_nhwc_f32
wasm-ld: error: libonnxruntime.a(conv_base.cc.o): undefined symbol: xnn_create_convolution2d_nhwc_f32
wasm-ld: error: libonnxruntime.a(conv_base.cc.o): undefined symbol: xnn_create_deconvolution2d_nhwc_qs8
wasm-ld: error: libonnxruntime.a(conv_base.cc.o): undefined symbol: xnn_create_convolution2d_nhwc_qs8
wasm-ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)

[raphaelmenges]

I could reproduce the issue in a workflow on GitHub actions:

-- Configuring done (17.6s)
CMake Error at /home/runner/work/onnxruntime-wasm-builds/onnxruntime-wasm-builds/build/Debug/_deps/googlexnnpack-src/CMakeLists.txt:748 (ADD_LIBRARY):
  Cannot find source file:

    /home/runner/work/onnxruntime-wasm-builds/onnxruntime-wasm-builds/build/Debug/_deps/googlexnnpack-src/src/amalgam/gen/scalar.c

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc

https://github.com/alfatraining/onnxruntime-wasm-builds/actions/runs/13069862337/job/36468940575#step:11:508

[guschmue]

we did not see much perf gain and use for xnnpack. We think spending some time to optimize wasm for mlas it the better choice but we have not gotten to that yet.

[raphaelmenges]

Thank you @guschmue for the feedback!

In terms of execution providers for the Web, the choices are a bit awkward. WebGL as the most stable one is now deprecated, the WebGPU one only works on Chromium with developer flags and Firefox Nightly so far and WebNN is not available in most Web browsers. I was counting on XNNPACK to get a better performance especially on Firefox, which is on CPU execution provider 2-3x slower than Chromium with the same task. Do you have any recommendation what to head for?

[guschmue]

webgpu is enabled on chromium since some time and stable on all platforms (linux it takes a few extra steps to use webgpu but it works well there too).
For Safari it is still under feature flag.
Firefox - yeah, that is a problem. If you use nightly, I have not been able to run lot of models.
For xnnpack we added support for a few ops and would need to add more but the perf gains have been less than we expected. Makes more sense for us to spend some time on mlas - few optimizations should go a long way there.
We have not gotten to it because the focus has been on webgpu, only that one will allow to run more costly models.
But spending some time on wasm optimization for mlas is still on the list.

[raphaelmenges]

webgpu is enabled on chromium since some time and stable on all platforms (linux it takes a few extra steps to use webgpu but it works well there too).

Cool! However, to my understanding it is not yet possible to compile onnxruntime as static library for the Web with WebGPU support: #23072 (comment)