Merge pull request #380 from gthiemonge/octavia_dcn

Octavia DCN support

DCN.md

@@ -0,0 +1,131 @@
+# Octavia DCN
+
+## Octavia in DCN deployments
+
+The deployment of the Octavia services in DCN differs from standard
+deployments.
+While it supports using only one Octavia management network across the
+Availability Zones for communication between the control plane and the Amphora
+instances, admins might want to isolate the network traffic and use one
+management network per AZ.
+
+In this case, they must configure the octavia-operator to define specific
+settings for those AZs.
+
+## Configuration of the Neutron AZs
+
+When deploying DCN, each compute node is assigned to an AZ (example: az[1..n]),
+the default AZ created for the control plane (az0 in this document) is not used
+by the compute nodes.
+It means that the `lb-mgmt-net` network created by the octavia-operator for the
+default AZ is not required.
+It can be (optionally) disabled by removing the route from the octavia Network
+Attachment Definition:
+
+Example:
+
+```shell
+oc edit network-attachment-definitions.k8s.cni.cncf.io octavia
+```
+
+```yaml
+spec:
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "octavia",
+      "type": "bridge",
+      "bridge": "octbr",
+      "ipam": {
+        "type": "whereabouts",
+        "range": "172.23.0.0/24",
+        "range_start": "172.23.0.30",
+        "range_end": "172.23.0.70"
+      }
+    }
+```
+
+The `lbMgmtNetwork.availabilityZones` spec of the Octavia Kind must contain the
+AZ of the control plane.
+
+The `lbMgmtNetwork.createDefaultLbMgmtNetwork` spec can be optionaly set to
+`false` to prevent the operator to create the default `lb-mgmt-net` network for
+default AZ.
+In this case, they should set `lbMgmtNetwork.lbMgmtRouterGateway` to an IP
+address of the octavia NAD, this address should be selected in a range that
+starts after the `ipam.range_end` IP address.
+
+Then `lbMgmtNetwork.availabilityZonesCIDRs` spec should define a different CIDR
+for each AZ. The octavia-operator will ensure that those CIDRs are routable from
+the Octavia service through a Neutron router.
+
+```shell
+oc patch openstackcontrolplane openstack-galera-network-isolation --type=merge --patch='
+    spec:
+      octavia:
+        template:
+          lbMgmtNetwork:
+            createDefaultLbMgmtNetwork: false
+            lbMgmtRouterGateway: 172.23.0.150
+            availabilityZones:
+            - az0
+            availabilityZoneCIDRs:
+              az1: 172.34.0.0/16
+              az2: 172.44.0.0/16
+'
+```
+
+With those settings, the octavia-operator will create:
+
+* a `lb-mgmt-az1-net` network with a `lb-mgmt-az1-subnet` subnet (CIDR
+  `172.34.0.0/16`) with availability_hints `az1`
+* a `lb-mgmt-az2-net` network with a `lb-mgmt-az2-subnet` subnet (CIDR
+  `172.44.0.0/16`) with availability_hints `az2`
+* an `octavia-provider-net` network with an `octavia-provider-subnet` subnet
+  (CIDR `172.23.0.0/24`)
+* an `octavia-link-router` router in `az0`, `az1` and `az2`,
+  `octavia-provider-subnet` is plugged into this router through a port with the
+  IP address `172.23.0.150`, `lb-mgmt-az1-subnet` and `lb-mgmt-az2-subnet` are
+  also plugged into the router
+
+## Creating Octavia Availability Zone Profiles and Availability Zones
+
+When creating a Load Balancer for a specific AZ in Octavia, some metadata must
+be passed to the Octavia service, to indicate which compute AZ and management network it should use to create Amphora VMs.
+
+Those metadata are stored in Octavia Availability Zone Profile and Availability
+Zones. They can be created by admins:
+
+```shell
+oc rsh openstackclient
+network_id=$(openstack network show -c id -f value lb-mgmt-az1-net)
+openstack loadbalancer availabilityzoneprofile create \
+    --provider amphora \
+    --availability-zone-data '{"compute_zone": "az1", "management_network": "'$network_id'"}' \
+    --name azp1
+openstack loadbalancer availabilityzone create \
+    --availabilityzoneprofile azp1 \
+    --name az1
+```
+
+```shell
+oc rsh openstackclient
+network_id=$(openstack network show -c id -f value lb-mgmt-az2-net)
+openstack loadbalancer availabilityzoneprofile create \
+    --provider amphora \
+    --availability-zone-data '{"compute_zone": "az2", "management_network": "'$network_id'"}' \
+    --name azp2
+openstack loadbalancer availabilityzone create \
+    --availabilityzoneprofile azp2 \
+    --name az2
+```
+
+A user can then pass an `availability-zone` parameter to the Octavia API when
+creating a Load Balancer
+
+```shell
+openstack loadbalancer create \
+    --availability-zone az2 \
+    --vip-subnet-id public-subnet \
+    --name lb1
+```

api/bases/octavia.openstack.org_octaviaamphoracontrollers.yaml

@@ -130,6 +130,11 @@ spec:
               octaviaProviderSubnetCIDR:
                 description: OctaviaProviderSubnetCIDR -
                 type: string
+              octaviaProviderSubnetExtraCIDRs:
+                description: OctaviaProviderSubnetExtraCIDRs -
+                items:
+                  type: string
+                type: array
               octaviaProviderSubnetGateway:
                 description: OctaviaProviderSubnetGateway -
                 type: string

api/bases/octavia.openstack.org_octavias.yaml

@@ -105,18 +105,39 @@ spec:
                 type: object
               lbMgmtNetwork:
                 default:
+                  createDefaultLbMgmtNetwork: true
                   manageLbMgmtNetworks: true
                 description: OctaviaLbMgmtNetworks Settings for Octavia management
                   networks
                 properties:
+                  availabilityZoneCIDRs:
+                    additionalProperties:
+                      type: string
+                    description: 'AvailabilityZoneCIDRs are the CIDRs of each management
+                      network associated with an Availability Zone (ex: {"az":"172.34.0.0/24",
+                      ...})'
+                    type: object
                   availabilityZones:
                     description: Availability zones for the octavia management network
                       resources
                     items:
                       type: string
                     type: array
+                  createDefaultLbMgmtNetwork:
+                    description: CreateDefaultLbMgmtNetwork - when True, octavia-operator
+                      creates a Management Network for the default Availability Zone
+                      of the control plane. Can be set to false when deploying OpenStack
+                      in DCN mode.
+                    type: boolean
+                  lbMgmtRouterGateway:
+                    description: LbMgmtRouterGateway is the IP address of the Octavia
+                      router on the Provider network, it's optional and used only
+                      when the routing informations are not passed through the Network
+                      Attachment Definition
+                    type: string
                   manageLbMgmtNetworks:
-                    default: true
+                    description: ManageLbMgmtNetworks - when True, octavia-operator
+                      creates the Neutron resources needed for its Management Network
                     type: boolean
                 type: object
               nodeSelector:
@@ -586,6 +607,11 @@ spec:
                   octaviaProviderSubnetCIDR:
                     description: OctaviaProviderSubnetCIDR -
                     type: string
+                  octaviaProviderSubnetExtraCIDRs:
+                    description: OctaviaProviderSubnetExtraCIDRs -
+                    items:
+                      type: string
+                    type: array
                   octaviaProviderSubnetGateway:
                     description: OctaviaProviderSubnetGateway -
                     type: string
@@ -787,6 +813,11 @@ spec:
                   octaviaProviderSubnetCIDR:
                     description: OctaviaProviderSubnetCIDR -
                     type: string
+                  octaviaProviderSubnetExtraCIDRs:
+                    description: OctaviaProviderSubnetExtraCIDRs -
+                    items:
+                      type: string
+                    type: array
                   octaviaProviderSubnetGateway:
                     description: OctaviaProviderSubnetGateway -
                     type: string
@@ -1122,6 +1153,11 @@ spec:
                   octaviaProviderSubnetCIDR:
                     description: OctaviaProviderSubnetCIDR -
                     type: string
+                  octaviaProviderSubnetExtraCIDRs:
+                    description: OctaviaProviderSubnetExtraCIDRs -
+                    items:
+                      type: string
+                    type: array
                   octaviaProviderSubnetGateway:
                     description: OctaviaProviderSubnetGateway -
                     type: string

api/v1beta1/amphoracontroller_types.go

@@ -151,6 +151,10 @@ type OctaviaAmphoraControllerSpecCore struct {
 	// +kubebuilder:validation:Optional
 	// OctaviaProviderSubnetCIDR -
 	OctaviaProviderSubnetCIDR string `json:"octaviaProviderSubnetCIDR"`
+
+	// +kubebuilder:validation:Optional
+	// OctaviaProviderSubnetExtraCIDRs -
+	OctaviaProviderSubnetExtraCIDRs []string `json:"octaviaProviderSubnetExtraCIDRs,omitempty"`
 }
 
 // OctaviaAmphoraControllerStatus defines the observed state of the Octavia Amphora Controller

api/v1beta1/conditions.go

@@ -40,6 +40,8 @@ const (
 	OctaviaAmphoraImagesReadyCondition condition.Type = "OctaviaAmphoraImagesReady"
 
 	OctaviaRsyslogReadyCondition condition.Type = "OctaviaRsyslogReady"
+
+	OctaviaManagementNetworkReadyCondition condition.Type = "OctaviaManagementNetworkReady"
 )
 
 // Common Messages used by API objects
@@ -130,4 +132,16 @@ const (
 
 	// OctaviaAmphoraImagesReadyCompleteMessage
 	OctaviaAmphoraImagesReadyCompleteMessage = "Octavia Amphora Images setup completed"
+
+	//
+	// OctaviaManagementNetworkReady condition messages
+	//
+	// OctaviaManagementNetworkReadyInitMessage
+	OctaviaManagementNetworkReadyInitMessage = "Octavia Management Network setup is not initialized"
+
+	// OctaviaManagementNetworkReadyErrorMessage
+	OctaviaManagementNetworkReadyErrorMessage = "Octavia Management Network setup error occured %s"
+
+	// OctaviaManagementNetworkReadyCompleteMessage
+	OctaviaManagementNetworkReadyCompleteMessage = "Octavia Management Network setup completed"
 )

api/v1beta1/octavia_types.go

@@ -171,7 +171,7 @@ type OctaviaSpecBase struct {
 	TenantName string `json:"tenantName"`
 
 	// +kubebuilder:validation:Optional
-	// +kubebuilder:default={manageLbMgmtNetworks: true}
+	// +kubebuilder:default={manageLbMgmtNetworks: true, createDefaultLbMgmtNetwork: true}
 	LbMgmtNetworks OctaviaLbMgmtNetworks `json:"lbMgmtNetwork"`
 
 	// +kubebuilder:validation:Optional
@@ -230,12 +230,28 @@ type PasswordSelector struct {
 // OctaviaLbMgmtNetworks Settings for Octavia management networks
 type OctaviaLbMgmtNetworks struct {
 	// +kubebuilder:validation:Optional
-	// +kubebuilder:default=true
-	ManageLbMgmtNetworks bool `json:"manageLbMgmtNetworks,omitempty"`
+	// ManageLbMgmtNetworks - when True, octavia-operator creates the Neutron resources needed for its Management Network
+	ManageLbMgmtNetworks bool `json:"manageLbMgmtNetworks"`
+
+	// +kubebuilder:validation:Optional
+	// CreateDefaultLbMgmtNetwork - when True, octavia-operator creates a
+	// Management Network for the default Availability Zone of the control
+	// plane. Can be set to false when deploying OpenStack in DCN mode.
+	CreateDefaultLbMgmtNetwork bool `json:"createDefaultLbMgmtNetwork"`
+
+	// +kubebuilder:validation:Optional
+	// LbMgmtRouterGateway is the IP address of the Octavia router on the
+	// Provider network, it's optional and used only when the routing
+	// informations are not passed through the Network Attachment Definition
+	LbMgmtRouterGateway string `json:"lbMgmtRouterGateway,omitempty"`
 
 	// +kubebuilder:validation:Optional
 	// Availability zones for the octavia management network resources
 	AvailabilityZones []string `json:"availabilityZones,omitempty"`
+
+	// +kubebuilder:validation:Optional
+	// AvailabilityZoneCIDRs are the CIDRs of each management network associated with an Availability Zone (ex: {"az":"172.34.0.0/24", ...})
+	AvailabilityZoneCIDRs map[string]string `json:"availabilityZoneCIDRs,omitempty"`
 }
 
 // OctaviaAmphoraFlavor Settings for custom Amphora flavors

config/crd/bases/octavia.openstack.org_octaviaamphoracontrollers.yaml

@@ -130,6 +130,11 @@ spec:
               octaviaProviderSubnetCIDR:
                 description: OctaviaProviderSubnetCIDR -
                 type: string
+              octaviaProviderSubnetExtraCIDRs:
+                description: OctaviaProviderSubnetExtraCIDRs -
+                items:
+                  type: string
+                type: array
               octaviaProviderSubnetGateway:
                 description: OctaviaProviderSubnetGateway -
                 type: string