Implement confidence intervals

[smalleni]

Resolves: #42

Confidence Interval improves with number of samples - example below

bash-3.2$ ./bin/${ARCH}/k8s-netperf podNetwork --iperf
INFO[2023-04-03 16:43:18] 📒 Reading netperf.yml file.                  
INFO[2023-04-03 16:43:19] 🔬 openshift-monitoring namespace found       
INFO[2023-04-03 16:43:19] Deploying in us-west-2a zone                 
INFO[2023-04-03 16:43:20] ♻️  Using existing Service                   
INFO[2023-04-03 16:43:20] ♻️  Using existing Service                   
INFO[2023-04-03 16:43:20] ♻️  Using existing Deployment                
INFO[2023-04-03 16:43:20] ⏰ Checking for client-across Pods to become ready... 
INFO[2023-04-03 16:43:20] ♻️  Using existing Deployment                
INFO[2023-04-03 16:43:20] ⏰ Checking for server Pods to become ready... 
INFO[2023-04-03 16:43:21] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:43:29] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:43:36] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:43:42] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
|    RESULT TYPE    | DRIVER  |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | MESSAGE SIZE | SAME NODE | DURATION | SAMPLES |     AVG VALUE      |    95% CONFIDENCE INTERVAL     |
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
| 📊 Stream Results | netperf | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 2       | 4931.600000 (Mb/s) | 4916.860803-4946.339197 (Mb/s) |
| 📊 Stream Results | iperf3  | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 2       | 2902.035968 (Mb/s) | 2565.730174-3238.341762 (Mb/s) |
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+
|        RESULT TYPE        | DRIVER  |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | MESSAGE SIZE | SAME NODE | DURATION | SAMPLES |  99%TILE VALUE  |
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+
| 📊 Stream Latency Results | netperf | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 2       | 4.000000 (usec) |
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+
(reverse-i-search)`vi': vi containers/Dockerfile 
bash-3.2$ vi netperf.yml 
bash-3.2$ ./bin/${ARCH}/k8s-netperf podNetwork --iperf
INFO[2023-04-03 16:44:23] 📒 Reading netperf.yml file.                  
INFO[2023-04-03 16:44:23] 🔬 openshift-monitoring namespace found       
INFO[2023-04-03 16:44:24] Deploying in us-west-2a zone                 
INFO[2023-04-03 16:44:25] ♻️  Using existing Service                   
INFO[2023-04-03 16:44:25] ♻️  Using existing Service                   
INFO[2023-04-03 16:44:25] ♻️  Using existing Deployment                
INFO[2023-04-03 16:44:25] ⏰ Checking for client-across Pods to become ready... 
INFO[2023-04-03 16:44:25] ♻️  Using existing Deployment                
INFO[2023-04-03 16:44:25] ⏰ Checking for server Pods to become ready... 
INFO[2023-04-03 16:44:26] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:44:34] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:44:41] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:44:49] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:44:57] 🗒️  Running netperf TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:45:05] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:45:10] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:45:16] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:45:22] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
INFO[2023-04-03 16:45:28] 🗒️  Running iperf3 TCP_STREAM (service false) for 5s 
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
|    RESULT TYPE    | DRIVER  |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | MESSAGE SIZE | SAME NODE | DURATION | SAMPLES |     AVG VALUE      |    95% CONFIDENCE INTERVAL     |
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
| 📊 Stream Results | netperf | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 5       | 4929.180000 (Mb/s) | 4919.596023-4938.763977 (Mb/s) |
| 📊 Stream Results | iperf3  | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 5       | 2807.974605 (Mb/s) | 2593.552710-3022.396500 (Mb/s) |
+-------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+--------------------+--------------------------------+
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+
|        RESULT TYPE        | DRIVER  |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | MESSAGE SIZE | SAME NODE | DURATION | SAMPLES |  99%TILE VALUE  |
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+
| 📊 Stream Latency Results | netperf | TCP_STREAM | 1           | false        | false   | 1024         | false     | 5        | 5       | 4.000000 (usec) |
+---------------------------+---------+------------+-------------+--------------+---------+--------------+-----------+----------+---------+-----------------+

[vishnuchalla]

Code changes look good to me. But looks like netperf and iperf differ by more than 1.5 times when it comes to throughput.

[jtaleric]

Code changes look good to me. But looks like netperf and iperf differ by more than 1.5 times when it comes to throughput.

Yeah, I noticed this at smaller packet sizes...

[jtaleric]

Code changes look good to me. But looks like netperf and iperf differ by more than 1.5 times when it comes to throughput.

Yeah, I noticed this at smaller packet sizes...

Seems relevant - esnet/iperf#1078

also, w/ jumbo frames, and smaller message sizes (1024 or really anything < 4096) w/ iperf3 we see a significant difference between netperf and iperf3.

[jtaleric]

Can you push another commit after running; go mod tidy; go mod vendor

[vishnuchalla]

Core logic looks good to me except for the workflow check failures.

[jtaleric]

Core logic looks good to me except for the workflow check failures.

locally it failed to build for me, i had to run go mod tidy; go mod vendor

20:44 $ make clean; make build
GOARCH=amd64 CGO_ENABLED=0 go build -v -mod vendor -o bin/amd64/k8s-netperf  ./cmd/k8s-netperf
pkg/results/result.go:16:2: cannot find package "." in:
	/home/jtaleric/code/k8s-netperf/vendor/github.com/aclements/go-moremath/stats
make: *** [Makefile:32: bin/amd64/k8s-netperf] Error 1
GOARCH=amd64 CGO_ENABLED=0 go build -v -mod vendor -o bin/amd64/k8s-netperf  ./cmd/k8s-netperf
pkg/results/result.go:16:2: cannot find package "." in:
	/home/jtaleric/code/k8s-netperf/vendor/github.com/aclements/go-moremath/stats
make: *** [Makefile:32: bin/amd64/k8s-netperf] Error 1

[rsevilla87]

@smalleni @jtaleric , some of the research on the iperf3 side about this in case you have interest:
esnet/iperf#1078 (comment)