[Feature]: NVIDIA Triton GenAI Perf Benchmark

[simon-mo]

🚀 The feature, motivation and pitch

The GenAI perf toolkit from NVIDIA can be used as an alternative benchmark tools for vLLM. While we already have benchmark scripts and framework in benchmarks directory, we should test out different load generators to compare the performance and accuracy of the benchmark clients.

In this issues, I described some tasks that we need help with to try out the new benchmark harness:

• Compare the output of the genai perf with the benchmark_serving, on the coverage of the result metrics and the accuracy.
• Vary the workloads ShareGPT/Sonnet/synthetics
• Implement it as an alternative harness through the script.

Happy to elaborate as well.

https://pypi.org/project/genai-perf/

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[catherinelee274]

would love to work on this. Will investigate the genai perf toolkit