Skip to content

Latest commit

 

History

History
365 lines (285 loc) · 14.1 KB

install-customization.md

File metadata and controls

365 lines (285 loc) · 14.1 KB

Assisted Installer Customization

Manifests

These APIs allows for adding arbitrary manifests to the set generated by openshift-install create manifests. A typical use case would be to create a MachineConfig which would make some persistent customization to a group of nodes. These will only take effect after the machine config operator is up and running so if a change is needed before or during the installation process, another API will be required.

Formats json, yaml, and multi-document yaml are accepted as manifests. Multi-document yaml manifests are split with unique names to avoid any collisions with existing files when they are added to their respective folders (either "manifests" or "openshift").

Yaml patches are also accepted and should be in the yaml-patch format, patches should end with .yml.patch, yaml.patch. Extensions may have a suffix of an underscore followed by some alphanumeric characters, for example .yml.patch_ocp_manifests or .yaml.patch_ocp_manifests Here is an example of the format of a Yaml patch file, for more information on the format, please see https://github.com/krishicks/yaml-patch

---
- op: replace
  path: /status/infrastructureTopology
  value: HighlyAvailable

Create a cluster manifest

Note: In environments where TLS is enabled (e.g. api.openshift.com), you will need to use the https scheme in URLs.

# base64 encoding of the example from https://docs.openshift.com/container-platform/4.6/installing/install_config/installing-customizing.html
content=YXBpVmVyc2lvbjogbWFjaGluZWNvbmZpZ3VyYXRpb24ub3BlbnNoaWZ0LmlvL3YxCmtpbmQ6IE1hY2hpbmVDb25maWcKbWV0YWRhdGE6CiAgbGFiZWxzOgogICAgbWFjaGluZWNvbmZpZ3VyYXRpb24ub3BlbnNoaWZ0LmlvL3JvbGU6IG1hc3RlcgogIG5hbWU6IDk5LW9wZW5zaGlmdC1tYWNoaW5lY29uZmlnLW1hc3Rlci1rYXJncwpzcGVjOgogIGtlcm5lbEFyZ3VtZW50czoKICAgIC0gJ2xvZ2xldmVsPTcnCg==
file=99-openshift-machineconfig-master-kargs.yaml
folder=openshift

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request POST \
    --data "{\"file_name\":\"$file\", \"folder\":\"$folder\", \"content\":\"$content\"}" \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests"

View a manifest’s contents

file=99-openshift-machineconfig-master-kargs.yaml
folder=openshift

curl --header "Authorization: Bearer $TOKEN" "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?file_name=$file&folder=$folder"

Manifests use cases

Configure storage on nodes using MachineConfig manifests

Changes to operating systems on OCP nodes can be done by creating what are referred to as MachineConfig objects that are managed by the Machine Config Operator (MCO).

The MachineConfig specification includes an ignition config for configuring the machines at first boot. This config object can be used to do things like modify files, systemd services, and other operating system features running on on OCP machines.

Specifically, the ignition config can be used to configure storage on the nodes (see ignition Config Spec for full details). The following MachineSet manifest example demonstrates how to add a partition to sda device on master nodes. I.e. apply the manifest before installation to have a partition named recovery of size 16 GiB on the master nodes.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 10-masters-storage-config
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      disks:
        - device: /dev/sda
          partitions:
            - label: recovery
              startMiB: 32768
              sizeMiB: 16384
      filesystems:
        - device: /dev/disk/by-partlabel/recovery
          label: recovery
          format: xfs

Discovery Ignition

The discovery ignition is used to make changes to the CoreOS live iso image which runs before we actually write anything to the target disk. An example use case would be configuring a separate container registry to pull the assisted-installer-agent image from.

The discovery ignition must use version 3.1.0 regardless of the version of the cluster that will eventually be created.

Patch the discovery ignition

# ignition patch file
{
  "ignition_config_override": "{\"ignition\": {\"version\": \"3.1.0\"}, \"storage\": {\"files\": [{\"path\": \"/etc/containers/registries.conf\", \"mode\": 420, \"overwrite\": true, \"user\": { \"name\": \"root\"},\"contents\": {\"source\": \"data:text/plain;base64,dW5xdWFsaWZpZWQtc2VhcmNoLXJlZ2lzdHJpZXMgPSBbInJlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tIiwgImRvY2tlci5pbyJdCltbcmVnaXN0cnldXQogICBwcmVmaXggPSAiIgogICBsb2NhdGlvbiA9ICJxdWF5LmlvL29jcG1ldGFsIgogICBtaXJyb3ItYnktZGlnZXN0LW9ubHkgPSBmYWxzZQogICBbW3JlZ2lzdHJ5Lm1pcnJvcl1dCiAgIGxvY2F0aW9uID0gImxvY2FsLnJlZ2lzdHJ5OjUwMDAvb2NwbWV0YWwiCg==\"}}]}}"
}

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data @discovery-ign.json \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID"

View the discovery ignition

curl --header "Authorization: Bearer $TOKEN" "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/'files?file_name=discovery.ign"

Install Config

These endpoints alter the default install config yaml used when running openshift-install create commands. Install config customization reference is available here Some of this content will be pinned to particular values, but most can be edited. Note, some of this content will be pinned to particular values by the assisted-installer (can't be overwritten).

You can compose the install-config overrides by creating a json string with the options you wish to set. An example install config with disabled hyperthreading for the control plane:

apiVersion: extensions.hive.openshift.io/v1beta1
baseDomain: example.com
controlPlane:
  name: master
  hyperthreading: Disabled
compute:
  - name: worker
    replicas: 5
metadata:
  name: test-cluster
platform: ...
pullSecret: '{"auths": ...}'
sshKey: ssh-ed25519 AAAA...

should look like this:

"{\"controlPlane\":{\"hyperthreading\":\"Disabled\"}}"

Patch the install config

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data '"{\"controlPlane\":{\"hyperthreading\":\"Disabled\"}}"' \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/install-config"

View the install config

curl --header "Authorization: Bearer $TOKEN" "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/install-config"

Exclude one or more optional components (capabilities)

Since OpenShift 4.12, it is possible to customize the install config to disable some components.

The features (baremetal-operator, marketplace, Console, Insights, Storage, CSISnapshot and openshift-samples-content) can be disabled by setting the baselineCapabilitySet and addittionalEnabledCapabilities parameters in the install-config.yaml.

Here is an example of disabling all capabilities except openshift-samples:

capabilities:
  baselineCapabilitySet: None
  additionalEnabledCapabilities:
    - openshift-samples

See https://docs.openshift.com/container-platform/latest/installing/cluster-capabilities.html for more information.

Important

Note that disabling capabilities could cause problems in some situations, e.g. disabling the baremetal-operator on a baremetal platform cluster will cause installation to fail.

Patching the install config

As with the parent example, install config can be patched with:

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data '"{\"capabilities\": {\"baselineCapabilitySet\": \"None\", \"additionalEnabledCapabilities\": [\"openshift-samples\"]}"' \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/install-config"

and the configuration verified via:

curl --header "Authorization: Bearer $TOKEN" "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/clusters/$CLUSTER_ID/install-config"

Pointer Ignition

The pointer ignition is used to customize the particular host when it reboots into the installed system. These changes will be made to the host, but will not be tracked as a machine config. Generally a machine config will be a better fit than editing the pointer ignition, but if some change is specific to an individual host it would be made here.

The pointer ignition override version must match the version of the ignition generated by openshift-installer. This means that the version required for the override will change depending on the cluster version being installed.

OCP Version(s) Ignition version
4.6 3.1.0
4.7, 4.8, 4.9, 4.10 3.2.0

Patch the pointer ignition

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data '{"config": "{\"ignition\": {\"version\": \"3.1.0\"}, \"storage\": {\"files\": [{\"path\": \"/etc/example\", \"contents\": {\"source\": \"data:text/plain;base64,SGVsbG8gZnJvbSBob3N0IDg0Njk2NzdiLThlZGEtNDQzOS1iNDQwLTc3ZGM5M2FkZmNlZgo=\"}}]}}"}' \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/ignition"

View the pointer ignition

curl --header "Authorization: Bearer $TOKEN" "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/$HOST_ID/ignition

Installer Params

This endpoint sets parameters to be passed to the coreos-installer command line in addition to the ones we provide by default. The only parameters that can be set with this endpoint are: ["--append-karg", "--delete-karg", "-n", "--copy-network", "--network-dir", "--save-partlabel", "--save-partindex", "--image-url"]

Set the installer params

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data '{"args": ["--append-karg", "nameserver=8.8.8.8", "-n"]}' \
"http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/installer-args"

Adding aditional trust bundle

When you need the cluster nodes / installation process to trust custom certificates, you can add them using the Assisted infra-env PATCH API's. The additional trust bundle can contain one or more CA certificates in the PEM format, for example:

-----BEGIN CERTIFICATE-----
...base-64-encoded
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...base-64-encoded
-----END CERTIFICATE-----

Following is an example of using the API. Begin by storing the certificates in a file called ca.pem, set TOKEN, ASSISTED_SERVICE_IP, ASSISTED_SERVICE_PORT and INFRA_ENV_ID to their appropriate values and run:

curl \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $TOKEN" \
    --request PATCH \
    --data "$( jq -n --arg BUNDLE "$(cat ca.pem)" '{ "additional_trust_bundle": $BUNDLE }' )" \
    "http://$ASSISTED_SERVICE_IP:$ASSISTED_SERVICE_PORT/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID"

Modifying Kernel Arguments for the Live ISO

Update the InfraEnv resource to modify the kernel arguments used in the live ISO (during the host discovery phase) . Currently, only the append (additional argument) operation is supported.

REST API

During InfraEnv Registeration

Note: additional parameters that are needed for InfraEnv creation are omitted from this example.

curl -X 'POST' \
  'http://api.openshift.com/api/assisted-install/v2/infra-envs' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
      "kernel_arguments": [
          {
            "operation": "append",
            "value": "audit=0"
          },
          {
            "operation": "append",
            "value": "trace=1"
          }
      ]
  }'

During InfraEnv Update

curl -X 'PATCH' \
    'http://api.openshift.com/api/assisted-install/v2/infra-envs/3fa85f64-5717-4562-b3fc-2c963f66afa6' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "kernel_arguments": [
          {
            "operation": "append",
            "value": "audit=0"
          },
          {
            "operation": "append",
            "value": "trace=1"
          }
        ]
    }'

When updating kernel arguments using this API call, all the kernel arguments are replaced. So the way to clear the kernel arguments is just to provide empty list:

curl -X 'PATCH' \
    'http://api.openshift.com/api/assisted-install/v2/infra-envs/3fa85f64-5717-4562-b3fc-2c963f66afa6' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "kernel_arguments": []
    }'

Kubernetes API

When using the Kubernetes API, the kernel arguments are set in the InfraEnv spec section:

apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: myinfraenv
  namespace: spoke-cluster
spec:
  kernelArguments:
    - operation: append
      value: audit=0
    - operation: append
      value: trace=1
  clusterRef:
    name: single-node
    namespace: spoke-cluster
  pullSecretRef:
    name: pull-secret