[Performance] Resize operation shows huge performance drop on Windows

[MatteoPagliani]

Describe the issue

Hi!

We observed a significant performance regression in the Resize operation when using CPUExecutionProvider on Windows. On Linux the behavior is the one expected. In particular, when opening the profile trace we realized that it is the Round node that shows the biggest performance drop between the two OS.

The mean latency of the Round node over 50 runs after warmup is ~6ms on Windows and ~0.1ms on Linux.

The same performance drop ratio appears disabling profiling and enabling all optimizations with GraphOptimizationLevel.ORT_ENABLE_ALL.

Model graph:

Windows profile:

Linux profile:

To reproduce

Here is the code used to reproduce the issue:

import torch
import torchvision.transforms as T
import onnxruntime as ort
import numpy as np
import time

class ResizeModel(torch.nn.Module):
    def __init__(self, size):
        super(ResizeModel, self).__init__()
        self.resize = T.Resize(size,
                               interpolation=T.InterpolationMode.BILINEAR,
                               antialias=False)

    def forward(self, x):
        return self.resize(x)


resize_model = ResizeModel(size=224)

dummy_input = torch.randint(0, 255, (1, 3, 390, 388), dtype=torch.uint8)

onnx_path = "resize_model.onnx"
resize_model.eval()
with torch.no_grad():
    torch.onnx.export(
        resize_model,
        (dummy_input,),
        onnx_path,
        input_names=["input"],
        output_names=["output"],
        opset_version=18
    )
print(f"Model exported to {onnx_path}")

print(ort.get_available_providers())
session_options = ort.SessionOptions()
session_options.enable_profiling = True
session_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
session = ort.InferenceSession(onnx_path,
                               session_options,
                               providers=["CPUExecutionProvider"])
onnx_input = {session.get_inputs()[0].name: dummy_input.numpy()}

# Warm-up runs
for _ in range(5):
    session.run(None, onnx_input)

# Benchmarking
latencies = []
for _ in range(50):
    start_time = time.time()
    session.run(None, onnx_input)
    latencies.append((time.time() - start_time) * 1000)  # Convert to milliseconds

# Compute mean latency
mean_latency = np.mean(latencies)
print(f"Mean latency over 50 runs: {mean_latency:.2f} ms")

Urgency

This slowdown severely impacts workloads that rely heavily on the Resize operator, particularly in image processing tasks.

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.20.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

No

[yuslepukhin]

Would you be comfortable testing perf in your environment if I shared changes on a branch with you?

Regression as compared to Linux or to some previous version?

[MatteoPagliani]

@yuslepukhin Yes please, feel free to share these changes. Regression is compared to Linux on the same machine (tested both in WSL Ubuntu and DevContainer Debian).

[yuslepukhin]

Most of the CPU time is taken by Resize though.

[MatteoPagliani]

@yuslepukhin I don't get it. As you can see from the profile the operation taking an anomalous amount of time in Windows is "/resize/Round". The execution time of "/resize/Resize" is approximately the same (~0.250 ms) in the two OS.

[yuslepukhin]

I am not seeing the same results as you do.
The overall model execution time is dominated by Resize kernel.
I am also not seeing significant differences in the Round kernel times.
 
I found on my devbox that the times swing widely even with the fixed input and the same number of threads that I tried to fix to 4 and then to 6 (thread settings are relevant for Resize).

This PR may make it better.

The statement below appears to contradict the title of the Issue.

The execution time of "/resize/Resize" is approximately the same (~0.250 ms) in the two OS.

[MatteoPagliani]

@yuslepukhin I changed the title of the issue since it could be misleading. What I meant is that the Resize operation (meaning the torchvision operation and not the actual ONNX node) shows a performance drop on Windows. Digging into the issue we realized that the actual ONNX node responsible for that slowdown is the Round node, which gets produced by the translation of the Resize operation into ONNX format. Now the issue should be more clear.

Thanks for providing the link to the PR, I will test it out. How eigen should make Windows and Linux performance comparable?