Skip to content

Latest commit

 

History

History
157 lines (128 loc) · 5.51 KB

policy-static-pools.md

File metadata and controls

157 lines (128 loc) · 5.51 KB

Static-Pools (STP) Policy

Overview

The static-pools (STP) builtin policy was inspired by CMK (CPU Manager for Kubernetes). It is an example policy demonstrating some capabilities of cri-resource-manager - not production ready.

Basically, the STP policy tries to replicate the functionality of the cmk isolate command CMK. The STP policy also has some compatibility features to function as close as possible drop-in replacement in order to allow easier testing and prototyping.

Features:

  • arbitrary number of configurable CPU list pools
  • dynamic configuration updates via cri-resmgr-agent

Please see the documentation of CMK for a more detailed description of the terminology and functionality.

CMK compatibility features:

  • supports the same environment variables as the original CMK, except for:
    • CMK_LOCK_TIMEOUT and CMK_PROC_FS: configuration variables that are not applicable in cri-resmgr context
    • CMK_LOG_LEVEL: not implemented, yet
    • CMK_NUM_CORES: not needed in cri-resmgr as we take this value directly from the container resource request
  • supports the existing configuration directory format of CMK for retrieving the pool configuration
  • parses the container command/args in an attempt to retrieve command line options of cmk isolate
  • supports generating CMK-specific node label and taint (off by default)

Deployment

Configuration

The STP policy tries to use configuration received from the cri-resmgr-agent which is, in turn, read from a Kubernetes ConfigMap.

As a fallback, the STP policy also supports configuration directory format of the original CMK. It tries to read the configuration from a location specified with -static-pools-conf-dir (/etc/cmk by default). Alternatively, you can provide a configuration file (YAML) by using the -static-pools-conf-file flag. See the example config for a starting point. However, if cri-resmgr at a later time receives a valid configuration from the cri-resmr-agent this will override the fallback configuration read from the directory or file.

NOTE: cri-resmgr does not have any utility for automatically generating a configuration. Thus, you need to either manually write one by yourself, or, run the cmk init command (of the original CMK) in order to create a legacy configuration directory structure.

Install cri-resmgr

Deploy cri-resmgr on each node as you would for any other policy.

Deploy Admission Webhook

You need to run and enable the cri-resmgr mutating admission webhook which is making resource request annotations. This is required so that the STP policy is able to inspect the extended resources (in this case, exclusive CPU cores) requested by containers. See the README for instructions.

Running cri-resmgr

An example of running cri-resmgr with the STP policy enabled, creating legacy CMK node label and taint:

cri-resmgr -policy static-pools -logger-debug stp -static-pools-create-cmk-node-label -static-pools-create-cmk-node-taint

Command Line Flags

Cri-resmgr has some command line flags specific to the STP policy:

  -static-pools-conf-dir string
        STP pool configuration directory (default "/etc/cmk")
  -static-pools-conf-file string
        STP pool configuration file
  -static-pools-create-cmk-node-label
        Create CMK-related node label for backwards compatibility
  -static-pools-create-cmk-node-taint
        Create CMK-related node taint for backwards compatibility

Running Workloads

The preferred way to specify the pod configuration is through environment variables. However, exclusive cores must be reserved by making a request of the cmk.intel.com/exclusive-cores extended resource. Naming of the extended resource has cmk prefix in order to provide backwards compatibility with the original CMK.

Pod Configuration Using Env Variables

The following environment variables are recognized:

Name Description
STP_NO_AFFINITY Do not set cpu affinity. The workload is responsible for reading the CMK_CPUS_ASSIGNED environment variable and setting the affinity itself.
STP_POOL Name of the pool to run in
STP_SOCKET_ID Socket where cores should be allocated. Set to -1 to accept any socket.

An example Pod spec for running a workload in the exclusive pool with one core reserved from socket id 0:

apiVersion: v1
kind: Pod
metadata:
  name: stp-test
spec:
  containers:
  - name: stp-test
    image: busybox
    env:
      - name: STP_POOL
        value: "exclusive"
      - name: STP_SOCKET_ID
        value: "0"
    command:
      - "sh"
      - "-c"
      - "while :; do echo ASSIGNED: $CMK_CPUS_ASSIGNED; sleep 1; done"
    resources:
      requests:
        cmk.intel.com/exclusive-cores: "1"
      limits:
        cmk.intel.com/exclusive-cores: "1"

Legacy CMK Pod Configuration

The STP policy tries to parse the container command/args in an attempt to retrieve the Pod configuration (from cmk isolate options). This is to provide backwards compatibility with existing CMK workload specs.

apiVersion: v1
kind: Pod
metadata:
  name: cmk-test
spec:
  containers:
  - name: cmk-test
    image: busybox
    command:
      - "sh"
      - "-c"
    args:
      - "/opt/bin/cmk isolate --conf-dir=/etc/cmk --pool=infra sleep 10000"