Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hugepages 2Mi and 1Gi fields to ResourceDescription and pass them to the statefulset #2311

Merged
merged 13 commits into from
Jan 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions charts/postgres-operator/crds/postgresqls.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,12 @@ spec:
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
# Note: the value specified here must not be zero or be higher
# than the corresponding limit.
hugepages-2Mi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-1Gi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
requests:
type: object
properties:
Expand All @@ -450,6 +456,12 @@ spec:
memory:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-2Mi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-1Gi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
schedulerName:
type: string
serviceAnnotations:
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/cluster_manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,14 @@ CPU and memory requests for the Postgres container.
memory requests for the Postgres container. Optional, overrides the
`default_memory_request` operator configuration parameter.

* **hugepages-2Mi**
hugepages-2Mi requests for the sidecar container.
Optional, defaults to not set.

* **hugepages-1Gi**
1Gi hugepages requests for the sidecar container.
Optional, defaults to not set.

### Limits

CPU and memory limits for the Postgres container.
Expand All @@ -371,6 +379,14 @@ CPU and memory limits for the Postgres container.
memory limits for the Postgres container. Optional, overrides the
`default_memory_limits` operator configuration parameter.

* **hugepages-2Mi**
hugepages-2Mi requests for the sidecar container.
Optional, defaults to not set.

* **hugepages-1Gi**
1Gi hugepages requests for the sidecar container.
Optional, defaults to not set.

## Parameters defining how to clone the cluster from another one

Those parameters are applied when the cluster should be a clone of another one
Expand Down Expand Up @@ -500,6 +516,14 @@ CPU and memory requests for the sidecar container.
memory requests for the sidecar container. Optional, overrides the
`default_memory_request` operator configuration parameter. Optional.

* **hugepages-2Mi**
hugepages-2Mi requests for the sidecar container.
Optional, defaults to not set.

* **hugepages-1Gi**
1Gi hugepages requests for the sidecar container.
Optional, defaults to not set.

### Limits

CPU and memory limits for the sidecar container.
Expand All @@ -512,6 +536,14 @@ CPU and memory limits for the sidecar container.
memory limits for the sidecar container. Optional, overrides the
`default_memory_limits` operator configuration parameter. Optional.

* **hugepages-2Mi**
hugepages-2Mi requests for the sidecar container.
Optional, defaults to not set.

* **hugepages-1Gi**
1Gi hugepages requests for the sidecar container.
Optional, defaults to not set.

## Connection pooler

Parameters are grouped under the `connectionPooler` top-level key and specify
Expand Down
21 changes: 21 additions & 0 deletions docs/user.md
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,27 @@ manifest the operator will raise the limits to the configured minimum values.
If no resources are defined in the manifest they will be obtained from the
configured [default requests](reference/operator_parameters.md#kubernetes-resource-requests).

### HugePages support
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this comes with K8s 1.28

Can we also link https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


The operator supports [HugePages](https://www.postgresql.org/docs/15/kernel-resources.html#LINUX-HUGEPAGES).
To enable HugePages, set the matching resource requests and/or limits in the manifest:

```yaml
spec:
resources:
requests:
hugepages-2Mi: 250Mi
hugepages-1Gi: 1Gi
limits:
hugepages-2Mi: 500Mi
hugepages-1Gi: 2Gi
```

There are no minimums or maximums and the default is 0 for both HugePage sizes,
but Kubernetes will not spin up the pod if the requested HugePages cannot be allocated.
For more information on HugePages in Kubernetes, see also
[https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/](https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/)

## Use taints, tolerations and node affinity for dedicated PostgreSQL nodes

To ensure Postgres pods are running on nodes without any other application pods,
Expand Down
4 changes: 4 additions & 0 deletions manifests/complete-postgres-manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,13 @@ spec:
requests:
cpu: 10m
memory: 100Mi
# hugepages-2Mi: 128Mi
# hugepages-1Gi: 1Gi
limits:
cpu: 500m
memory: 500Mi
# hugepages-2Mi: 128Mi
# hugepages-1Gi: 1Gi
patroni:
failsafe_mode: false
initdb:
Expand Down
12 changes: 12 additions & 0 deletions manifests/postgresql.crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,12 @@ spec:
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
# Note: the value specified here must not be zero or be higher
# than the corresponding limit.
hugepages-2Mi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-1Gi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
requests:
type: object
properties:
Expand All @@ -448,6 +454,12 @@ spec:
memory:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-2Mi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
hugepages-1Gi:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
schedulerName:
type: string
serviceAnnotations:
Expand Down
21 changes: 19 additions & 2 deletions pkg/apis/acid.zalan.do/v1/crds.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@ package v1
import (
"fmt"

acidzalando "github.com/zalando/postgres-operator/pkg/apis/acid.zalan.do"
"github.com/zalando/postgres-operator/pkg/util"
apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

acidzalando "github.com/zalando/postgres-operator/pkg/apis/acid.zalan.do"
"github.com/zalando/postgres-operator/pkg/util"
)

// CRDResource* define names necesssary for the k8s CRD API
Expand Down Expand Up @@ -684,6 +685,14 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
"hugepages-2Mi": {
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
"hugepages-1Gi": {
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
},
},
"requests": {
Expand All @@ -697,6 +706,14 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
"hugepages-2Mi": {
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
"hugepages-1Gi": {
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
},
},
},
Expand Down
6 changes: 4 additions & 2 deletions pkg/apis/acid.zalan.do/v1/postgresql_type.go
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,10 @@ type PostgresqlParam struct {

// ResourceDescription describes CPU and memory resources defined for a cluster.
type ResourceDescription struct {
CPU string `json:"cpu"`
Memory string `json:"memory"`
CPU string `json:"cpu"`
Memory string `json:"memory"`
HugePages2Mi string `json:"hugepages-2Mi"`
HugePages1Gi string `json:"hugepages-1Gi"`
}

// Resources describes requests and limits for the cluster resouces.
Expand Down
20 changes: 17 additions & 3 deletions pkg/cluster/k8sres.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ import (
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/intstr"

"golang.org/x/exp/maps"
batchv1 "k8s.io/api/batch/v1"
"k8s.io/apimachinery/pkg/labels"

acidv1 "github.com/zalando/postgres-operator/pkg/apis/acid.zalan.do/v1"
"github.com/zalando/postgres-operator/pkg/spec"
"github.com/zalando/postgres-operator/pkg/util"
Expand All @@ -28,9 +32,6 @@ import (
"github.com/zalando/postgres-operator/pkg/util/k8sutil"
"github.com/zalando/postgres-operator/pkg/util/patroni"
"github.com/zalando/postgres-operator/pkg/util/retryutil"
"golang.org/x/exp/maps"
batchv1 "k8s.io/api/batch/v1"
"k8s.io/apimachinery/pkg/labels"
)

const (
Expand Down Expand Up @@ -267,6 +268,19 @@ func fillResourceList(spec acidv1.ResourceDescription, defaults acidv1.ResourceD
}
}

if spec.HugePages2Mi != "" {
requests[v1.ResourceHugePagesPrefix+"2Mi"], err = resource.ParseQuantity(spec.HugePages2Mi)
if err != nil {
return nil, fmt.Errorf("could not parse hugepages-2Mi quantity: %v", err)
}
}
if spec.HugePages1Gi != "" {
requests[v1.ResourceHugePagesPrefix+"1Gi"], err = resource.ParseQuantity(spec.HugePages1Gi)
if err != nil {
return nil, fmt.Errorf("could not parse hugepages-1Gi quantity: %v", err)
}
}

return requests, nil
}

Expand Down
128 changes: 126 additions & 2 deletions pkg/cluster/k8sres_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ import (
"fmt"
"reflect"
"sort"
"time"

"testing"
"time"

"github.com/stretchr/testify/assert"

Expand Down Expand Up @@ -2979,6 +2978,131 @@ func TestGenerateResourceRequirements(t *testing.T) {
ResourceLimits: acidv1.ResourceDescription{CPU: "1", Memory: "2Gi"},
},
},
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we amend the other test to test for no huge pages set? Given its K8s 1.28 feature lets make sure its not set if not enabled.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a test to cover this

subTest: "test HugePages are not set on container when not requested in manifest",
config: config.Config{
Resources: configResources,
PodManagementPolicy: "ordered_ready",
},
pgSpec: acidv1.Postgresql{
ObjectMeta: metav1.ObjectMeta{
Name: clusterName,
Namespace: namespace,
},
Spec: acidv1.PostgresSpec{
Resources: &acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{},
ResourceLimits: acidv1.ResourceDescription{},
},
TeamID: "acid",
Volume: acidv1.Volume{
Size: "1G",
},
},
},
expectedResources: acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{
CPU: "100m",
Memory: "100Mi",
},
ResourceLimits: acidv1.ResourceDescription{
CPU: "1",
Memory: "500Mi",
},
},
},
{
subTest: "test HugePages are passed through to the postgres container",
config: config.Config{
Resources: configResources,
PodManagementPolicy: "ordered_ready",
},
pgSpec: acidv1.Postgresql{
ObjectMeta: metav1.ObjectMeta{
Name: clusterName,
Namespace: namespace,
},
Spec: acidv1.PostgresSpec{
Resources: &acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{
HugePages2Mi: "128Mi",
HugePages1Gi: "1Gi",
},
ResourceLimits: acidv1.ResourceDescription{
HugePages2Mi: "256Mi",
HugePages1Gi: "2Gi",
},
},
TeamID: "acid",
Volume: acidv1.Volume{
Size: "1G",
},
},
},
expectedResources: acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{
CPU: "100m",
Memory: "100Mi",
HugePages2Mi: "128Mi",
HugePages1Gi: "1Gi",
},
ResourceLimits: acidv1.ResourceDescription{
CPU: "1",
Memory: "500Mi",
HugePages2Mi: "256Mi",
HugePages1Gi: "2Gi",
},
},
},
{
subTest: "test HugePages are passed through on sidecars",
config: config.Config{
Resources: configResources,
PodManagementPolicy: "ordered_ready",
},
pgSpec: acidv1.Postgresql{
ObjectMeta: metav1.ObjectMeta{
Name: clusterName,
Namespace: namespace,
},
Spec: acidv1.PostgresSpec{
Sidecars: []acidv1.Sidecar{
{
Name: "test-sidecar",
DockerImage: "test-image",
Resources: &acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{
HugePages2Mi: "128Mi",
HugePages1Gi: "1Gi",
},
ResourceLimits: acidv1.ResourceDescription{
HugePages2Mi: "256Mi",
HugePages1Gi: "2Gi",
},
},
},
},
TeamID: "acid",
Volume: acidv1.Volume{
Size: "1G",
},
},
},
expectedResources: acidv1.Resources{
ResourceRequests: acidv1.ResourceDescription{
CPU: "100m",
Memory: "100Mi",
HugePages2Mi: "128Mi",
HugePages1Gi: "1Gi",
},
ResourceLimits: acidv1.ResourceDescription{
CPU: "1",
Memory: "500Mi",
HugePages2Mi: "256Mi",
HugePages1Gi: "2Gi",
},
},
},
}

for _, tt := range tests {
Expand Down
Loading