Add automated benchmark #43
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nirs
force-pushed
the
bench
branch
11 times, most recently
from
ec74fcd to
fd230ce
Compare
nirs
force-pushed
the
bench
branch
3 times, most recently
from
a88d55f to
8f28f1a
Compare
Previous version became too complicated and buggy. Clean up by extracting a vment package with small modules. The example contains now connecting the helper and vm. The details of how to create the helper and the vm are in the vmnet package.
nirs
force-pushed
the
bench
branch
4 times, most recently
from
7edd40f to
ef5d924
Compare
It is useful to benchmark a local build of a driver for testing fixes or
performance improvements in the driver. We can do this by building the
driver locally and specifying the local build using
--driver-command /path/to/local/driver
The bench tools manages testing vms, run benchmarks, and generate plots
from the benchmarks results.
Usage:
% ./bench -h
usage: bench [-h] {create,run,plot,delete} ...
options:
-h, --help show this help message and exit
commands:
{create,run,plot,delete}
create Create benchmark vms
run Run benchmarks
plot Generate plots
delete Delete benchmark vms
% ./bench run -h
usage: bench run [-h] filename
positional arguments:
filename Benchmark file
options:
-h, --help show this help message and exit
% ./bench plot -h
usage: bench plot [-h] filename
positional arguments:
filename Plot file
options:
-h, --help show this help message and exit
Instead of documenting the manual commands, show how to run the new bench tool.
And remove duplicate and partial docs from the example script.
The same plot configuration can be used with multiple benchmarks results directories from different machines.
When testing single network (e.g vmnet-helper), there is not need to include the name in the label. This gives more space for the data and avoids issues with data labels and very short bars.
When testing the same vm count, don't include the number of vms in the labels. This gives more room to the data and avoids issues with data labels on very short bars.
The benchmark runs krunkit without offloading and vfkit with offloading, to show the effect of offloading in both drivers. This benchmark requires building libkrun with a patch disabling offloading and build krunkit with the local libkrun.
Change the plots to focus on one theme: - plots/drivers.yaml: Compare builtin drivers, including krunkit with offloading since it is the current release, and the numbers are awesome. This is probably the best plot for the readme. - plots/offloading.yaml: Compare krunkit with other drivers with and without offloading with other builtin drivers. This is focused on making offloading configurable in libkrun, and maybe improve host-to-vm performance. - plots/vmnet-helper-vs-socket_vmnet.yaml: Compare vmnet-helper to socket_vment with 2 vms. This is relevant to lima and minikube developers, helping to integrate vmnet-helper in lima and minikube. - plots/socket_vmnet-vms.yaml: Compare socket_vmnet performance with different number of vms. This is relevant mostly to socket_vmnet developer.
Collect the results from all tests, and plot them using one barh() per test. This shows all test in the same scale, making it easy to understand the performance of the test configurations in all tests. The socket_vmnet plot was separated to vz and qemu plots, since showing 6 configurations for 3 tests is too much. This makes it harder to compare vz and qemu performance, but the main point of that plot is how adding vms decrease the performance.
Including results from: - benchmarks/full.yaml - benchmarks/offloading.yaml - socket_vmnet test/perf.sh - plots from plots/*.yaml
Same configuration tested on M2.
nirs
force-pushed
the
bench
branch
8 times, most recently
from
fbb00cd to
78b3b84
Compare
Replace the tables with the most important plots and refer to the performance directory for more info. Tart and softnet results were dropped, since I did not compare them in the last benchmarks. We can results later when we have some automation to run the tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #41
Fixes #42
Fixes #37
Fixes #47