postgres.kubedb.com/postgres-cluster 16.1 is stuck on Provisioning status and never getting ready

[salcsal]

I am trying to run postgres ha in AWS eks cluster - everything is ok except pods are not accepting any database connection.

Here is my postgres-cluster.yaml

apiVersion: kubedb.com/v1alpha2
kind: Postgres
metadata:
name: postgres-cluster
namespace: demo
spec:
version: "16.1"
replicas: 3
standbyMode: Hot
storageType: Durable
storage:
storageClassName: "efs-sc"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
terminationPolicy: WipeOut

salcsal@boss1:~/kube-db$ kubectl get all -n demo
NAME READY STATUS RESTARTS AGE
pod/postgres-cluster-0 2/2 Running 0 77m
pod/postgres-cluster-1 2/2 Running 0 77m
pod/postgres-cluster-2 2/2 Running 0 77m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres-cluster ClusterIP 10.100.61.251 5432/TCP,2379/TCP 77m
service/postgres-cluster-pods ClusterIP None 5432/TCP,2380/TCP,2379/TCP 77m
service/postgres-cluster-standby ClusterIP 10.100.246.125 5432/TCP 77m

NAME READY AGE
statefulset.apps/postgres-cluster 3/3 77m

NAME TYPE VERSION AGE
appbinding.appcatalog.appscode.com/postgres-cluster kubedb.com/postgres 16.1 77m

NAME VERSION STATUS AGE
postgres.kubedb.com/postgres-cluster 16.1 Provisioning 77m
salcsal@boss1:~/kube-db$

salcsal@boss1:/kube-db$ kubectl get pvc -n demo
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-postgres-cluster-0 Bound pvc-78f18cb2-8c24-4348-9dc5-7e55f6b92978 1Gi RWX efs-sc 79m
data-postgres-cluster-1 Bound pvc-d7e1ccde-873b-422a-8861-db200f76a374 1Gi RWX efs-sc 79m
data-postgres-cluster-2 Bound pvc-04053293-879b-410d-9901-0ef4ed2d7510 1Gi RWX efs-sc 79m
salcsal@boss1:/kube-db$

Describe shows The Postgres: demo/postgres-cluster is not accepting client requests. error: failed to run query: dial tcp 10.100.61.251:5432: connect: connection refused

salcsal@boss1:/kube-db$ kubectl -n demo describe postgres.kubedb.com/postgres-cluster
Name: postgres-cluster
Namespace: demo
Labels:
Annotations:
API Version: kubedb.com/v1alpha2
Kind: Postgres
Metadata:
Creation Timestamp: 2024-08-03T20:31:40Z
Finalizers:
kubedb.com
Generation: 2
Resource Version: 1580050
UID: 23ac328c-3487-48dc-a0d0-daea4d295d9d
Spec:
Allowed Schemas:
Namespaces:
From: Same
Arbiter:
Resources:
Limits:
Memory: 256Mi
Requests:
Cpu: 200m
Memory: 256Mi
Storage: 2Gi
Auth Secret:
Name: postgres-cluster-auth
Auto Ops:
Client Auth Mode: md5
Coordinator:
Resources:
Limits:
Memory: 256Mi
Requests:
Cpu: 200m
Memory: 256Mi
Security Context:
Allow Privilege Escalation: false
Capabilities:
Drop:
ALL
Run As Group: 70
Run As Non Root: true
Run As User: 70
Seccomp Profile:
Type: RuntimeDefault
Health Checker:
Failure Threshold: 1
Period Seconds: 10
Timeout Seconds: 10
Leader Election:
Election Tick: 10
Heartbeat Tick: 1
Maximum Lag Before Failover: 67108864
Period: 300ms
Transfer Leadership Interval: 1s
Transfer Leadership Timeout: 1m0s
Pod Template:
Controller:
Metadata:
Spec:
Affinity:
Pod Anti Affinity:
Preferred During Scheduling Ignored During Execution:
Pod Affinity Term:
Label Selector:
Match Labels:
app.kubernetes.io/instance: postgres-cluster
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: postgreses.kubedb.com
Namespaces:
demo
Topology Key: kubernetes.io/hostname
Weight: 100
Pod Affinity Term:
Label Selector:
Match Labels:
app.kubernetes.io/instance: postgres-cluster
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: postgreses.kubedb.com
Namespaces:
demo
Topology Key: topology.kubernetes.io/zone
Weight: 50
Container Security Context:
Allow Privilege Escalation: false
Capabilities:
Drop:
ALL
Run As Group: 70
Run As Non Root: true
Run As User: 70
Seccomp Profile:
Type: RuntimeDefault
Init Containers:
Name: postgres-init-container
Resources:
Limits:
Memory: 512Mi
Requests:
Cpu: 200m
Memory: 256Mi
Security Context:
Allow Privilege Escalation: false
Capabilities:
Drop:
ALL
Run As Group: 70
Run As Non Root: true
Run As User: 70
Seccomp Profile:
Type: RuntimeDefault
Resources:
Limits:
Memory: 1Gi
Requests:
Cpu: 500m
Memory: 1Gi
Security Context:
Fs Group: 70
Run As Group: 70
Run As User: 70
Service Account Name: postgres-cluster
Replicas: 3
Replication:
Wal Keep Size: 1024
Wal Limit Policy: WALKeepSize
Ssl Mode: disable
Standby Mode: Hot
Storage:
Access Modes:
ReadWriteMany
Resources:
Requests:
Storage: 1Gi
Storage Class Name: efs-sc
Storage Type: Durable
Termination Policy: WipeOut
Version: 16.1
Status:
Conditions:
Last Transition Time: 2024-08-03T20:31:40Z
Message: The KubeDB operator has started the provisioning of Postgres: demo/postgres-cluster
Reason: DatabaseProvisioningStartedSuccessfully
Status: True
Type: ProvisioningStarted
Last Transition Time: 2024-08-03T20:31:55Z
Message: All desired replicas are ready.
Reason: AllReplicasReady
Status: True
Type: ReplicaReady
Last Transition Time: 2024-08-03T20:32:05Z
Message: The Postgres: demo/postgres-cluster is not accepting client requests. error: failed to run query: dial tcp 10.100.61.251:5432: connect: connection refused
Observed Generation: 2
Reason: DatabaseNotAcceptingConnectionRequest
Status: False
Type: AcceptingConnection
Last Transition Time: 2024-08-03T20:32:05Z
Message: The Postgres: demo/postgres-cluster is not ready.
Observed Generation: 2
Reason: ReadinessCheckFailed
Status: False
Type: Ready
Observed Generation: 1
Phase: Provisioning
Events:
salcsal@boss1:/kube-db$

Any help will be appreciated.

Thanks,

Salil Chattopadhyay
[email protected]

[souravbiswassanto]

Hello @salcsal, Sorry for the late.
Can you give me the log from postgres an pg-coordinator container from all the pods?

• kubectl logs -f -n demo postgres-cluster-0 -c postgres
• kubectl logs -f -n demo postgres-cluster-0 -c pg-coordinator
• kubectl logs -f -n demo postgres-cluster-1 -c postgres
• kubectl logs -f -n demo postgres-cluster-1 -c pg-coordinator
• kubectl logs -f -n demo postgres-cluster-2 -c postgres
• kubectl logs -f -n demo postgres-cluster-2 -c pg-coordinator