investigate possibly reporting actual CPU usage

[rptaylor]

Currently KAPEL reports walltime * cores. The APEL specifications about "cpu time" are not very clear. But it could be useful to report actual CPU usage efficiency.

There are Prometheus metrics such as:
container_cpu_user_seconds_total{namespace="harvester",pod="grid-job-3156500-cxlh5"}

This reports duplicate records for a pod:

container_cpu_user_seconds_total{container="job-container", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3774a4_abb2_4e93_ae3b_23750b385538.slice/cri-containerd-d1c6f999a3cea33e45c27730f7a202a6ff026b2aa159758961c30a358b482d28.scope", image="git.computecanada.ca:4567/rptaylor/misc/atlas-grid-almalinux9-localuser:20240124", instance="10.5.7.79:10250", job="kubelet", name="d1c6f999a3cea33e45c27730f7a202a6ff026b2aa159758961c30a358b482d28", namespace="harvester", node="kermes-dev-k8s-node-a03", pod="grid-job-3156500-cxlh5", service="kube-prometheus-kubelet"}
	67274.08
container_cpu_user_seconds_total{endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3774a4_abb2_4e93_ae3b_23750b385538.slice", instance="10.5.7.79:10250", job="kubelet", namespace="harvester", node="kermes-dev-k8s-node-a03", pod="grid-job-3156500-cxlh5", service="kube-prometheus-kubelet"}
	67273.34
container_cpu_user_seconds_total{endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3774a4_abb2_4e93_ae3b_23750b385538.slice/cri-containerd-44c2af6f208a48575595e9d78cb34defc01a45e8bcea8b6c3a7737ed7b72456c.scope", image="registry.k8s.io/pause:3.6", instance="10.5.7.79:10250", job="kubelet", name="44c2af6f208a48575595e9d78cb34defc01a45e8bcea8b6c3a7737ed7b72456c", namespace="harvester", node="kermes-dev-k8s-node-a03", pod="grid-job-3156500-cxlh5", service="kube-prometheus-kubelet"}
	0.01

Another challenge would be container-level accounting and querying rather than pod-level, in particular in cases of multi-container pods, with each container requesting different resources.

Further related info:

• https://stackoverflow.com/questions/34923788/prometheus-convert-cpu-user-seconds-to-cpu-usage
• https://devcodef1.com/news/1011742/prometheus-cpu-utilization-per-pod-per-node

[rptaylor]

Implemented in #54