IssueRaiser

EKS nodes lose readiness when containers exhaust memory

<!-- Please use this template while reporting a bug and provide as much info as possible. Please also search for existing open and closed issues that may answer your question. Thanks!--> **What happened**: When our applications consume too much memory, K8s nodes on EKS clusters lose readiness and become completely inoperable for extended periods of time. This means that instead of being rescheduled immediately, pods remain stuck in a pending state, resulting in noticeable downtime. This does not happen on GKE clusters. **What you expected to happen**: Nodes should never lose readiness, instead the containers should be restarted and/or the pods should be `OOMKilled`. **How to reproduce it (as minimally and precisely as possible)**: 1. Provision a single node EKS cluster running an EC2 instance type of `m5.large`. 1. Apply the following deployment resource. ``` apiVersion: apps/v1 kind: Deployment metadata: annotations: labels: app: stress-ng name: stress-ng spec: selector: matchLabels: app: stress-ng template: metadata: labels: app: stress-ng spec: containers: - args: - -c - stress-ng --bigheap 0 command: - /bin/bash image: alexeiled/stress-ng:latest-ubuntu name: stress-ng ``` 1. Observe the node lose readiness. ``` $ k get nodes NAME

Posted by dasrirez on Jan 9, 2023, 9:10 PM

Tags

Related issues

Raise real-world problems, find people facing the same issue, and discuss solutions.2,500 issues0 comments

github (SaaS)

EKS nodes lose readiness when containers exhaust memory

dasrirezJan 9, 2023, 9:10 PM

<!-- Please use this template while reporting a bug and provide as much info as possible. Please also search for existing open and closed issues that may answer your question. Thanks!--> **What happened**: When our applications consume too much memory, K8s nodes on EKS clusters lose readiness and become completely inoperable for extended periods of time. This means that instead of being rescheduled immediately, pods remain stuck in a pending state, resulting in noticeable downtime. This does not happen on GKE clusters. **What you expected to happen**: Nodes should never lose readiness, instead the containers should be restarted and/or the pods should be `OOMKilled`. **How to reproduce it (as minimally and precisely as possible)**: 1. Provision a single node EKS cluster running an EC2 instance type of `m5.large`. 1. Apply the following deployment resource. ``` apiVersion: apps/v1 kind: Deployment metadata: annotations: labels: app: stress-ng name: stress-ng spec: selector: matchLabels: app: stress-ng template: metadata: labels: app: stress-ng spec: containers: - args: - -c - stress-ng --bigheap 0 command: - /bin/bash image: alexeiled/stress-ng:latest-ubuntu name: stress-ng ``` 1. Observe the node lose readiness. ``` $ k get nodes NAME

Comments (0)

No comments yet. Start the discussion.

Comment on this issue