Skip to content

Commit

Permalink
Clarify the docs of resource usage
Browse files Browse the repository at this point in the history
  • Loading branch information
wzshiming committed Mar 28, 2024
1 parent 5c13cee commit 4c56cdc
Show file tree
Hide file tree
Showing 10 changed files with 138 additions and 93 deletions.
55 changes: 36 additions & 19 deletions site/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -170,15 +170,22 @@ menu:
- identifier: platform-specific-binaries
pageRef: "/docs/user/kwokctl-platform-specific-binaries"
parent: kwokctl-advanced-usage
- identifier: with-argo
pageRef: "/docs/examples/argo"

- identifier: examples
title: Examples
parent: kwokctl-advanced-usage
- identifier: with-linkerd
- identifier: argo
pageRef: "/docs/examples/argo"
parent: examples
- identifier: linkerd
pageRef: "/docs/examples/linkerd"
parent: kwokctl-advanced-usage
- identifier: with-istio
parent: examples
- identifier: istio
pageRef: "/docs/examples/istio"
parent: kwokctl-advanced-usage
parent: examples
- identifier: metrics-server
pageRef: "/docs/examples/metrics-server"
parent: examples

- identifier: tools
title: Tools
Expand All @@ -200,33 +207,43 @@ menu:
- identifier: api-reference
pageRef: "/docs/generated/apis"
parent: configuration
- identifier: stages
- identifier: stages-configuration
pageRef: "/docs/user/stages-configuration"
parent: configuration
- identifier: port-forward
pageRef: "/docs/user/port-forward-configuration"

- identifier: pod-interaction
title: Pod Interaction
parent: configuration
- identifier: exec
- identifier: port-forward-configuration
pageRef: "/docs/user/port-forward-configuration"
parent: pod-interaction
- identifier: exec-configuration
pageRef: "/docs/user/exec-configuration"
parent: configuration
- identifier: logs
parent: pod-interaction
- identifier: logs-configuration
pageRef: "/docs/user/logs-configuration"
parent: configuration
- identifier: attach
parent: pod-interaction
- identifier: attach-configuration
pageRef: "/docs/user/attach-configuration"
parent: configuration
- identifier: metrics
parent: pod-interaction
- identifier: metrics-configuration
pageRef: "/docs/user/metrics-configuration"
parent: configuration
- identifier: resource-usage
params:
collapse: never
- identifier: resource-usage-configuration
pageRef: "/docs/user/resource-usage-configuration"
parent: metrics-configuration

- identifier: extensions
title: Extensions
parent: configuration
- identifier: go-template
pageRef: "/docs/user/go-template"
parent: configuration
parent: extensions
- identifier: cel-expressions
pageRef: "/docs/user/cel-expressions"
parent: configuration
parent: extensions

# Design Children
- identifier: introduction
Expand Down
2 changes: 1 addition & 1 deletion site/content/en/docs/examples/argo.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "With Argo"
title: "Argo"
---

# Argo
Expand Down
2 changes: 1 addition & 1 deletion site/content/en/docs/examples/istio.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "With Istio"
title: "Istio"
---

# Istio
Expand Down
2 changes: 1 addition & 1 deletion site/content/en/docs/examples/linkerd.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "With Linkerd"
title: "Linkerd"
---

# Linkerd
Expand Down
38 changes: 38 additions & 0 deletions site/content/en/docs/examples/metrics-server.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "Metrics Server"
---

# Metrics Server

More information about an be found at [Metrics] and [ResourceUsage].

## Fetch default metrics usage

``` bash
wget https://github.com/kubernetes-sigs/kwok/releases/download/v0.5.1/metrics-usage.yaml
```

## Set up Cluster

``` bash
kwokctl create cluster --enable-metrics-server -c metrics-usage.yaml
```

## Create Node and Pod

``` bash
kwokctl scale node --replicas 2
kwokctl scale pod --replicas 8
```

## Test Metrics

Wait about 45 seconds for the metrics server to collect the data.

``` bash
kubectl top node
kubectl top pod
```

[Metrics]: {{< relref "/docs/user/metrics-configuration" >}}
[ResourceUsage]: {{< relref "/docs/user/resource-usage-configuration" >}}
4 changes: 2 additions & 2 deletions site/content/en/docs/user/cel-expressions.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "CEL Expressions in `kwok`"
title: "CEL Expressions"
---

# Notes on CEL Expressions in `kwok`
# CEL Expressions in `kwok`

The page provides a concise note on writing CEL expressions in `kwok` CRs.

Expand Down
9 changes: 3 additions & 6 deletions site/content/en/docs/user/go-template.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
---
title: "Go Template in `kwok`"
title: "Go Template"
---

# Notes on Go Template in `kwok`

# Go Template in `kwok`

The page provides a concise note on writing go templates in kwok CRs.

Expand All @@ -15,12 +14,10 @@ You must follow [the go text template syntax] when writing the templates.
For predefined functions of go text template, please refer to [go text template functions].
Besides the built-in functions, `kwok` also supports [sprig template functions].

It is worth noting that the "context" (which is denoted by the period character `.` ) to a template in `kwok` is set to the
It is worth noting that the "context" (which is denoted by the period character `.`) to a template in `kwok` is set to the
referenced Kubernetes resource.
For example, you can use `.metadata.name` in a template to obtain the corresponding Kubernetes resource name.



[the go text template syntax]: https://pkg.go.dev/text/template
[go text template functions]: https://pkg.go.dev/text/template#hdr-Functions
[sprig template functions]: https://masterminds.github.io/sprig/
52 changes: 9 additions & 43 deletions site/content/en/docs/user/metrics-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,68 +41,34 @@ spec:
There are total four metric-related endpoints in kubelet: `/metrics`, `/metrics/resource`, `/metrics/probe` and `/metrics/cadvisor`,
all of which are exposed with a Prometheus style. The Metrics resource is capable of simulating endpoints with such style.

To simulate a metric endpoint, first, you need to specify the RESTful `path` of the endpoint,
which will be installed and exposed by the metric service of `kwok` at port `10247` after applied.
The `path` must start with `/metrics`, otherwise, `kwok` will not install it.
The `path` field is required and must start with `/metrics`.
To distinguish the metrics of different nodes, the path includes a variable `{nodeName}` that is replaced by the node name.


{{< hint "info" >}}
Starting from metrics-server 0.7.0, it is allowed to specify the path to scrape metrics for a node.
Specifically, metrics-server will check if a node has annotation `metrics.k8s.io/resource-metrics-path`
and use it as the target metric scrape path.
Combined with the Metric CR, the feature makes it possible to integrate `kwok` and metrics-server.
For a fake node, by adding that annotation and setting its value to the `path`
specified in a Metric resource, metrics-server will collect data from the endpoints exposed by `kwok` instead of
scrapping from kubelet.
{{< /hint >}}

Besides, compared to kubelet, which only exposes the metric of the node it is located on, `kwok` needs to expose the
metrics of all the fake nodes it manages. Instead of creating a separate Metric CR for each fake node, it is possible
to bind all the metrics endpoints from different nodes into a single `path`. Metric CR allows for a built-in
`{nodeName}` path parameter to be included in the `path` field. For example: `/metrics/nodes/{nodeName}/metrics/resource`.
With `{nodeName}`, a single `path` is able to differentiate the metric data from different nodes.


The `metrics` field are used to customize the return body of the installed metrics endpoint.

The descriptions of each sub-field are available at [Metric API][Metric].
The descriptions of each sub-field are available at [Metric API][Metrics].
For readers' convenience, we also mirror the documents here with some additional notes.

`metrics` is a list of specific configuration items, with each corresponding to a Prometheus style metric:
* `name` defines the metric name.
* `labels` defines the metric labels, with each item corresponding to a specific metric label.
- `name` is a const string that provides the label name.
- `value` is represented as a CEL expression that dynamically determines the label value.
- `value` is represented as a [CEL expressions] that dynamically determines the label value.
For example: you can use `node.metadata.name` to reference the node name as the label value.
* `help` defines the help string of a metric.
* `kind` defines the type of the metric: `counter`, `gauge`, or `histogram`.
* `dimension` defines where the data comes from. It could be `node`, `pod`, or `container`.
* `value` is a CEL expression that defines the metric value if `kind` is `counter` or `gauge`.
Please refer to [CEL expressions in `kwok`] for more detailed instructions that might be helpful to simulate the metric value.
* `value` is a [CEL expressions] that defines the metric value if `kind` is `counter` or `gauge`.
* `buckets` is exclusively for customizing the data of the metric of kind `histogram`.
- `le`, which defines the histogram bucket’s upper threshold, has the same meaning as the one of Prometheus histogram bucket.
That is, each bucket contains values less than or equal to `le`.
- `value` is a CEL expression that provides the value of the bucket.
- `hidden` indicates whether to show the bucket in the metric.
But the value of the bucket will be calculated and cumulated into the next bucket.

Please refer to [Metrics for kubelet's "metrics/resource" endpoint][metrics resource endpoint] for a detailed example.


## Out-of-box Metric Config

`kwok` currently provides the [Metrics config][metrics resource endpoint] that is capable of
simulating kubelet's `"metrics/resource"` endpoint.

To integrate the simulated endpoint with metrics-server (required version >= 0.7.0), add the
`"metrics.k8s.io/resource-metrics-path": "/metrics/nodes/<nodeName>/metrics/resource"` annotation to the fake
nodes managed by `kwok`.

<img width="700px" src="/img/demo/resource-usage.svg">
## Examples

Please refer to [Metrics for kubelet's `/metrics/resource` endpoint][ResourceUsage] for a detailed.

[configuration]: {{< relref "/docs/user/configuration" >}}
[Metrics]: {{< relref "/docs/generated/apis" >}}#kwok.x-k8s.io/v1alpha1.Metrics
[CEL expressions in `kwok`]: {{< relref "/docs/user/cel-expressions" >}}
[metrics resource endpoint]: https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/metrics/resource
[ResourceUsage (ClusterResourceUsage)]: {{< relref "/docs/user/resource-usage-configuration" >}}
[CEL expressions]: {{< relref "/docs/user/cel-expressions" >}}
[ResourceUsage]: {{< relref "/docs/user/resource-usage-configuration" >}}
65 changes: 46 additions & 19 deletions site/content/en/docs/user/resource-usage-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ This document walks you through how to simulate the resource usage of pod(s).

{{< /hint >}}

## What is ResourceUsage?
## What is a ResourceUsage?

[ResourceUsage] is a [`kwok` Configuration][configuration] that allows users to define and simulate the resource usages of a single pod.
The [ResourceUsage] is a [`kwok` Configuration][configuration] that allows users to define and simulate the resource usages of a single pod.

The YAML below shows all the fields of a ResourceUsage resource:

Expand Down Expand Up @@ -47,8 +47,8 @@ If `containers` is not given in a group, the `usage` in that group will be appli
{{< /hint >}}

You can simply set a static [Quantity value] (`100Mi`, `1000m`, etc.) via `cpu.value` and `memory.value` to define the cpu and memory resource usage respectively.
Besides, users are also allowed to provide a [CEL expression] via `expressions` to describe the resource usage more flexibly. For example,
the following expression tries to extract the cpu resource usage from the pod's annotation if it has or use a default value.
Besides, users are also allowed to provide a [CEL expression] via `expressions` to describe the resource usage more flexibly.
For example, the following expression tries to extract the cpu resource usage from the pod's annotation if it has or use a default value.

```yaml
expression: |
Expand All @@ -67,8 +67,7 @@ yields memory usage that grows linearly with time.
```yaml
expression: (pod.SinceSecond() / 60.0) * Quantity("1Mi")
```
Please refer to [CEL expressions in `kwok`] for an exhausted list that may be helpful to configure dynamic resource usage.

Please refer to [CEL expressions in `kwok`][CEL expressions] for an exhausted list that may be helpful to configure dynamic resource usage.

### ClusterResourceUsage

Expand Down Expand Up @@ -108,28 +107,56 @@ ClusterResourceUsage has an additional `selector` field for specifying the targe

The `usages` field of ClusterResourceUsage has the same semantic with the one in ResourceUsage.

Please refer to [pod resource usage from annotation] for a concrete example.
## Dependencies

- [Metrics] and [`/metrics/resource` endpoint][metrics resource endpoint]
- metrics-server version >= 0.7.0
it will get the annotation `metrics.k8s.io/resource-metrics-path` for the node instead of the default `/metrics/resource` endpoint.

## Where to get the simulation data?
### Where to get the simulation data?

The resource usages defined in ResourceUsage and ClusterResourceUsage resources can be fetched from the metric service of `kwok` at port `10247` with path `/metrics/nodes/{nodeName}/metrics/resource`,
where `{nodeName}` is the name of the fake node that the pod is scheduled to.
The returned metrics are similar to the response from kubelet's `/metrics/resource` endpoint.
The data can be fetched from the metric service of `kwok` with the path `/metrics/nodes/{nodeName}/metrics/resource`,
where `{nodeName}` is the name of the fake node.
The metrics returned are similar to the response from kubelet's `/metrics/resource` endpoint.

Please refer to [`kwok` Metric][Metric] about how to integrate `kwok` simulated metrics endpoints with metrics-server.
Please refer to [`kwok` Metrics][Metrics] about how to integrate `kwok` simulated metrics endpoints with metrics-server.

## Dependencies
## Out-of-the-box

ResourceUsage or ClusterResourceUsage only takes effect when the [Metric] feature is also enabled and
[the default Metric resource] that simulates kubelet's `/metrics/resource` endpoint is applied.
Currently, a configuration is provided to quickly simulate the resource usage of pods.

- [Metrics resource endpoint][metrics resource endpoint]
- [Resource usage from annotation][resource usage from annotation]

```yaml
# the <nodeName> should be replaced with the name of the node
kind: Node
apiVersion: v1
metadata:
annotations:
metrics.k8s.io/resource-metrics-path: "/metrics/nodes/<nodeName>/metrics/resource"
...
```

```yaml
# specify the resource usage of a pod via annotation
kind: Pod
apiVersion: v1
metadata:
annotations:
kwok.x-k8s.io/usage-cpu: "100m"
kwok.x-k8s.io/usage-memory: "100Mi"
...
```

<img width="700px" src="/img/demo/resource-usage.svg">

[configuration]: {{< relref "/docs/user/configuration" >}}
[ResourceUsage]: {{< relref "/docs/generated/apis" >}}#kwok.x-k8s.io/v1alpha1.ResourceUsage
[ClusterResourceUsage]: {{< relref "/docs/generated/apis" >}}#kwok.x-k8s.io/v1alpha1.ClusterResourceUsage
[Quantity value]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes
[CEL expression]: https://github.com/google/cel-spec/blob/master/doc/langdef.md
[Metric]: {{< relref "/docs/user/metrics-configuration" >}}
[the default Metric resource]: https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/metrics/resource
[pod resource usage from annotation]: https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/metrics/usage/usage-from-annotation.yaml
[CEL expressions in `kwok`]: {{< relref "/docs/user/cel-expressions" >}}
[Metrics]: {{< relref "/docs/user/metrics-configuration" >}}
[metrics resource endpoint]: https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/metrics/resource/metrics-resource.yaml
[resource usage from annotation]: https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/metrics/usage/usage-from-annotation.yaml
[CEL expressions]: {{< relref "/docs/user/cel-expressions" >}}
2 changes: 1 addition & 1 deletion site/content/en/docs/user/stages-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This document walks you through how to configure the Stages of Lifecycle.

## What is a Stage?

The [Stage API] is a [`kwok` Configuration][configuration] that allows users to define and simulate different stages in the lifecycle of Kubernetes resources, such as nodes and pods.
The [Stage] is a [`kwok` Configuration][configuration] that allows users to define and simulate different stages in the lifecycle of Kubernetes resources, such as nodes and pods.
Each Stage resource specifies a `resourceRef` field that identifies the type of resource that the stage applies to, and a `selector` field that determines when the stage should be executed.

A Stage resource has the following fields:
Expand Down

0 comments on commit 4c56cdc

Please sign in to comment.