Add automated benchmark

.gitignore

@@ -8,3 +8,9 @@
 # Editor files
 /.zed/
 /pyrightconfig.json
+
+# Python
+__pycache__
+
+# macOS metadata
+.DS_Store

README.md

@@ -221,11 +221,12 @@ them in your application logs or redirect them to a file.
 The example tool shows how to integrate vmnet-helper with *vfkit* or
 *qemu*.
 
-To install all requirements for creating virtual machine using *vfkit*
-and *qemu* run:
+To install the requirements for creating virtual machine using *vfkit*
+*krunkit*, and *qemu* run:
 
 ```console
-brew install python3 vfkit qemu cdrtools
+brew tap slp/krunkit
+brew install python3 vfkit krunkit qemu cdrtools
 python3 -m venv .venv
 source .venv/bin/activate
 pip install pyyaml
@@ -249,136 +250,100 @@ To stop the virtual machine and the vmnet-helper press *Control+C*.
 
 ## Performance
 
-vmnet-helper is up to *10.4 times faster* compared to
-[socket_vmnet](https://github.com/lima-vm/socket_vmnet) using Apple
-Virtualization framework, and up to *3.0 times faster* compared to
-[softnet](https://github.com/cirruslabs/softnet).
-
-With [krunkit](https://github.com/containers/krunkit) vm to vm
-performance is similar to [tart](https://tart.run/), using native
-bridged network.
-
-### iMac M3
-
-Results from iMac M3 running macOS 15.2, testing only shared mode
-(192.168.105/24).
-
-| network      | mode   | vm          | host to vm     | cpu  | vm to vm      | cpu  |
-|--------------|--------|-------------|----------------|------|---------------|------|
-| vmnet-helper | shared | vfkit       |   13.0 Gbits/s |  61% |  17.2 Gbits/s |  69% |
-| vmnet-helper | shared | qemu        |    3.0 Gbits/s |  23% |   2.9 Gbits/s |  31% |
-| softnet      | shared | tart        |    9.5 Gbits/s |  97% |   5.7 Gbits/s |  90% |
-| socket_vmnet | shared | vz          |    3.9 Gbits/s |  81% |   1.9 Gbits/s |  91% |
-| socket_vmnet | shared | qemu        |    3.8 Gbits/s |  41% |   2.5 Gbits/s |  78% |
-
-### MacBook Pro M2 Max
-
-Results from MacBook Pro M2 Max running macOS 15.2, testing both shared
-and bridged modes.
-
-Using iperf3 `--length 1m` option.
-
-| network      | mode    | vm         | host to vm     | cpu  | vm to vm      | cpu  |
-|--------------|---------|------------|----------------|------|---------------|------|
-| vmnet-helper | shared  | vfkit      |   10.5 Gbits/s |  71% |  13.5 Gbits/s |  88% |
-| vmnet-helper | bridged | vfkit      |   13.0 Gbits/s |  88% |  12.9 Gbits/s |  86% |
-| vmnet-helper | shared  | krunkit[1] |    1.4 Gbits/s |  33% |  29.9 Gbits/s |  63% |
-| vmnet-helper | bridged | krunkit[1] |    1.4 Gbits/s |  34% |  31.0 Gbits/s |  63% |
-| vmnet-helper | shared  | krunkit[2] |    9.8 Gbits/s |  70% |   9.4 Gbits/s |  84% |
-| vmnet-helper | bridged | krunkit[2] |    9.9 Gbits/s |  94% |   8.9 Gbits/s |  88% |
-| vmnet-helper | shared  | qemu       |    2.7 Gbits/s |  20% |   2.4 Gbits/s |  35% |
-| softnet      | shared  | tart       |    5.5 Gbits/s |  98% |   5.4 Gbits/s | 100% |
-| vz           | bridged | tart       |    5.3 Gbits/s |    - |  35.9 Gbits/s |    - |
-| socket_vmnet | shared  | vz         |    2.5 Gbits/s |  95% |   1.3 Gbits/s | 130% |
-| socket_vmnet | shared  | qemu       |    2.8 Gbits/s |  41% |   1.4 Gbits/s | 104% |
-
-Notes:
-- [1] krunkit built with libkrun upstream
-- [2] krunkit built with libkrun patched to disable offloading
+We benchmarked vmnet-helper with 3 VMs types (vfkit, krunkit, qemu) in
+all operation modes supported by the vmnet framework (shared, bridged,
+host), in 3 directions (host to vm, vm to host, vm to vm), on 2 machines
+(iMac M3, MacBook Pro M2 Max) running macOS 15.3.1.
 
-## Performance testing
+See the [performance](/performance) directory for full test results.
 
-### host to vm
+### Comparing to socket_vment
 
-Running iperf3-darwin client on the host, and iperf3 server in the
-virtual machine.
+Comparing to [socket_vmnet](https://github.com/lima-vm/socket_vmnet)
+with [lima](https://github.com/lima-vm/lima) using VZ and qemu vm types,
+vmnet-helper with [vfkit](https://github.com/crc-org/vfkit) is up to *10
+times faster*, and vmnet-helper with [qemu](https://www.qemu.org/) is up
+to *3 times faster*.
 
-server vm:
+![vmnet-helper vs socket_vmnet - bridged network](performance/M3/plot/vmnet-helper-vs-socket_vmnet/bridged.png)
 
-```console
-iperf3 -s
-```
+### Comparing different VMs
+
+Performance depends on VM type and transfer direction.
+[vfkit](https://github.com/crc-org/vfkit) shows consistent performance
+in all tests.  [krunkit](https://github.com/containers/krunkit) is up to
+*3 times faster* than vfkit in vm to host test, but also up to *10 times
+slower* in host to vm test. [qemu](https://www.qemu.org/) is up to *5
+times slower* than vfkit.
+
+![vmnet-helper drivers - bridged network](performance/M3/plot/drivers/bridged.png)
+
+## Performance testing
 
-host:
+To install the requirements for running benchmarks and generating plots
+run:
 
 ```console
-iperf3-darwin -c {server-vm-ip} -t 30
+source .venv/bin/activate
+pip install matplotlib
 ```
 
-### vm to vm
+### Running benchmarks
 
-Running iperf3 client in client virtual machine, and iperf3 server on
-the server virtual machine.
+Create vms for benchmarking:
 
-server vm:
-
-```console
-iperf3 -s
+```
+./bench create
 ```
 
-client vm:
+To run all benchmarks with all drivers and all operation modes and store
+iperf3 results in json format use:
 
-```console
-iperf3 -c {server-vm-ip} -t 30
+```
+./bench run benchmarks/full.yaml
 ```
 
-### vmnet-helper
+The benchmark results are stored under `out/bench/vmnet-helper`.
 
-Created server and client vms for *vfkit* and *qemu* drivers.
+See the [benchmarks](/benchmarks) directory for additional configurations.
 
-vfkit:
+When done you can delete the vms using:
 
-```console
-./example server --driver vfkit &
-./example client --driver vfkit &
+```
+./bench delete
 ```
 
-qemu:
+### Creating plots
+
+To create plots from benchmark results run:
 
-```console
-./example server --driver qemu &
-./example client --driver qemu &
 ```
+./bench plot -o out plots/drivers.yaml
+```
+
+The plots use the results stored under `out/bench` and created under
+`out/plot`.
+
+See the [plots](/plots) directory for additional configurations.
 
 ### socket_vmnet
 
 Running socket_vmnet as launchd service, creating virtual machines with
-lima 1.0.3.
+lima 1.0.6.
 
 Tests run using socket_vmnet `test/perf.sh` script:
 
 ```console
 test/perf.sh create
-test/perf.sh host-to-vm
-test/perf.sh vm-to-vm
+test/perf.sh run
 ```
 
-Testing *qemu* by editing lima.yaml and adding `vmType: qemu`.
+To include socket_vment results in the plots copy the test results to
+the output directory:
 
-### softnet
-
-Created ubuntu server and client vm using:
-
-```console
-tart clone ghcr.io/cirruslabs/ubuntu:latest server
-tart set server --cpu 1 --memory 2048
-tart clone ghcr.io/cirruslabs/ubuntu:latest client
-tart set client --cpu 1 --memory 2048
-tart run --net-softnet --net-softnet-allow 0.0.0.0/0 server &
-tart run --net-softnet --net-softnet-allow 0.0.0.0/0 client &
 ```
-
-The vms are using Ubuntu 22.04 LTS. Other vms are using Ubuntu 24.10.
+cp ~/src/socket_vmnet/test/perf.out/socket_vment out/bench/
+```
 
 ## Similar tools