[Bug]: Primary PostgreSQL Instance Not Gracefully Shutting Down and Missing WAL File
### Is there an existing issue already for this bug? - [X] I have searched for an existing issue, and could not find anything. I believe this is a new bug. ### I have read the troubleshooting guide - [X] I have read the troubleshooting guide and I think this is a new bug. ### I am running a supported version of CloudNativePG - [X] I have read the troubleshooting guide and I think this is a new bug. ### Contact Details oren@aperio.ai ### Version 1.22.0 ### What version of Kubernetes are you using? 1.27 ### What is your Kubernetes environment? Cloud: Amazon EKS ### How did you install the operator? Helm ### What happened? We deployed the operator using a Helm chart and the PostgreSQL cluster resource using a separate Helm chart. Three instances of PostgreSQL were deployed on EC2 Spot instances. After a few hours, it appears that the primary instance (postgres-1) did not gracefully shut down, leading to the absence of the relevant Write-Ahead Logging (WAL) file. Subsequently, attempts to use pg_rewind have failed, with the process encountering an empty file. Additionally, the file "pg_xact/0000" is also empty, contributing to the inability to resolve the issue and leaving the old primary instance in a stuck state. ### Cluster resource ```shell apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: annotations: meta.helm.sh/release-name: postgres-cn meta.helm.sh/release-namespace: stg creationTimestamp: "2024-01-09T12:06:49Z" generation: 4 l
Posted by orenzah on Jan 10, 2024, 1:24 PM
Tags
Related issues
- ⚠️ PHP CVEs. Update required!
- I think RTS could continue to work if someone made a massive leap of faith and put all the modern infra around it that a AAA extraction s...
- RFC: Pluggable SessionDB Provider — PostgreSQL, MySQL, and Beyond
- v2.1.113+ broken on Termux/Android: native binary requires glibc, no JS fallback
- I’m firmly anti-Tahoe and haven’t updated, and I’ve started to find Apple despicable.
- [FEATURE] Link a source control repo as the source for organization skills
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
reliability (Sysadmin)
⚠️ PHP CVEs. Update required!
I think RTS could continue to work if someone made a massive leap of faith and put all the modern infra around it that a AAA extraction s...
RFC: Pluggable SessionDB Provider — PostgreSQL, MySQL, and Beyond
v2.1.113+ broken on Termux/Android: native binary requires glibc, no JS fallback
I’m firmly anti-Tahoe and haven’t updated, and I’ve started to find Apple despicable.
[FEATURE] Link a source control repo as the source for organization skills
Feature Request: Option to disable paste text collapse in input field
We went down the WASM sandboxing rabbit hole at Gobii when building our agent infra.
[Bug]: Primary PostgreSQL Instance Not Gracefully Shutting Down and Missing WAL File
### Is there an existing issue already for this bug? - [X] I have searched for an existing issue, and could not find anything. I believe this is a new bug. ### I have read the troubleshooting guide - [X] I have read the troubleshooting guide and I think this is a new bug. ### I am running a supported version of CloudNativePG - [X] I have read the troubleshooting guide and I think this is a new bug. ### Contact Details oren@aperio.ai ### Version 1.22.0 ### What version of Kubernetes are you using? 1.27 ### What is your Kubernetes environment? Cloud: Amazon EKS ### How did you install the operator? Helm ### What happened? We deployed the operator using a Helm chart and the PostgreSQL cluster resource using a separate Helm chart. Three instances of PostgreSQL were deployed on EC2 Spot instances. After a few hours, it appears that the primary instance (postgres-1) did not gracefully shut down, leading to the absence of the relevant Write-Ahead Logging (WAL) file. Subsequently, attempts to use pg_rewind have failed, with the process encountering an empty file. Additionally, the file "pg_xact/0000" is also empty, contributing to the inability to resolve the issue and leaving the old primary instance in a stuck state. ### Cluster resource ```shell apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: annotations: meta.helm.sh/release-name: postgres-cn meta.helm.sh/release-namespace: stg creationTimestamp: "2024-01-09T12:06:49Z" generation: 4 l
Comments (0)
No comments yet. Start the discussion.