Pods evicted without waiting for terminationGracePeriodSeconds due to lack resources of Kubernetes nodes

kubernetes

If you specify only requests resources of memory or ephemeral-storage that includes emptyDir volume, pods can consume resources more than settings, and kubelet monitors following signals and may evict them to avoid running out of node resources.

  • memory.available
  • nodefs.available
  • nodefs.inodesFree
  • imagefs.available
  • imagefs.inodesFree
  • pid.available

These thresholds can be set using evictionSoft and evictionHard in KubeletConfiguration, and if the soft value is reached, there is an eviction-max-pod-grace-period until it is evicted, but if the hard value is also reached, pods are immediately evicted. TerminationGracePeriodSeconds of pod is not considered in any case.

Pods that will be evicted are chosen from ones using resources more than requested, and Pod Priority is also considered.

If you use EKS AMI, evictionHard is set in bootstrap.sh, which runs at startup.

$ cat /etc/kubernetes/kubelet/kubelet-config.json | jq ".evictionHard"
{
  "memory.available": "100Mi",
  "nodefs.available": "10%",
  "nodefs.inodesFree": "5%"
}