Integrate KleidiAI for MatMulNBits via MlasQNBitGemm #23627
Conversation
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
| @@ -99,6 +100,10 @@ function(setup_mlas_source_for_windows) | |||
| ${MLAS_SRC_DIR}/halfgemm_kernel_neon_fp16.cpp | |||
| ) | |||
|
|
|||
| setup_kleidiai() | |||
There was a problem hiding this comment.
It means that kleidiai will be a new dependency for all ONNX Runtime build configs. For such changes the onnx runtime team needs to hold an internal discussion with the leadership of this project.
|
Please fix the iOS build errors. |
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
|
/azp run Linux CPU CI Pipeline, Windows CPU CI Pipeline, Linux QNN CI Pipeline |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
Can the workflows be retriggered please? |
|
/azp run Linux CPU CI Pipeline, Windows CPU CI Pipeline, Linux QNN CI Pipeline |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
|
Can the workflows be retriggered please? |
|
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Linux CPU CI Pipeline |
|
/azp run Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline |
|
/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,CoreML CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline |
|
Azure Pipelines successfully started running 6 pipeline(s). |
|
Azure Pipelines successfully started running 10 pipeline(s). |
1 similar comment
|
Azure Pipelines successfully started running 10 pipeline(s). |
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Co-authored-by: Edward Chen <[email protected]>
Co-authored-by: Edward Chen <[email protected]>
Signed-off-by: Michael Tyler <[email protected]>
Description
This PR integrates Arm® KleidiAI™ to provide optimized assembly kernels for matrix multiplication with 4-bit quantized weights. These changes target the MlasQNBitGemm functions, and can be utilized via the MatMulNBits operator.
Motivation and Context
These optimized assembly kernels lead to significant performance improvements on Arm-based devices.