DevOps Diaries

Hey — It's Avinash Tietler 👋

Here you get a use cases,top news, tools, and articles from the DevOps.

IN TODAY'S EDITION

Use Case

• Troubleshoot and Fix in Kubernetes - Node Not Ready

🚀 Top News

Microsoft, OpenAI have their own secret version of what AGI means

👀 Remote Jobs

Coffeee.io is hiring for Devops Engineer(Remote)

📚️ Resources

110+ Hours of Free LinkedIn Learning Courses with Free Certification

USE CASE

Troubleshoot and Fix in Kubernetes - Node Not Ready

The Node Not Ready error in Kubernetes indicates a situation where a node within a Kubernetes cluster is not in a healthy state to accept pods for execution. This status is a crucial indicator for cluster administrators, as it signifies that the Kubernetes scheduler will not assign new pods to the affected node until it returns to a Ready state. 

_{The ^{Node Not Ready} status in Kubernetes indicates that a node in your cluster is not functioning properly and cannot host any pods.}

Nodes may enter a Not Ready state for a variety of reasons, ranging from network issues, resource exhaustion, misconfigurations, or underlying hardware problems. Understanding and resolving the root cause of this error is essential to maintain the operational efficiency and reliability of a Kubernetes cluster.

In Kubernetes, Nodes can be in one of several states, reflecting their current status and ability to accept workloads:

• A node in the Ready state is healthy and capable of accepting new pods. 

• A node in the Not Ready state has encountered an issue that prevents it from functioning correctly. 

• The Unknown state indicates that the Kubernetes master has lost communication with the node, and its status cannot be determined.

To determine if a node is experiencing a Node Not Ready error, and obtain the information necessary to solve the problem, follow these steps:

Troubleshooting the error

1. Checking Node State

The first step is to check the state of the nodes in the cluster. This can be done using the kubectl get nodes command, which lists all nodes and their statuses. A node marked as NotReady requires further investigation to understand the underlying issues.

2. Obtaining Node Details

kubectl describe node <node-name>

command provides comprehensive details about the node, including its conditions, events, and configuration. This information is useful for diagnosing the root cause of the Not Ready status, offering insights into any errors or warnings that the node might be experiencing. Analyzing this output helps pinpoint specific issues, guiding the troubleshooting and resolution processes.

Here are a few things to notice in the output, which could indicate the cause of the problem:

• Conditions section: This section lists various node health indicators.

• Events section: This section records significant events in the life of the node.

3. Checking System Logs

Logs from the kubelet, the primary component running on each node that communicates with the Kubernetes master, can provide insights into any errors or issues it is encountering. 

You can access kubelet logs using journalctl or other logging utilities, depending on the node’s operating system:

Possible Causes of the Node Not Ready Error

There are several conditions that can result in a node having a Not Ready status.

Possible Causes

1. Node Issues:

   • The kubelet service is not running.

   • Node has insufficient resources (CPU, memory, or disk).

   • Node has networking or connectivity issues.

   • Node is under maintenance or power-off.

2. API Server Issues:

   • The node cannot communicate with the Kubernetes API server.

3. Component Issues:

   • Missing or misconfigured critical components such as kubelet, container runtime (e.g., Docker, containerd), or kube-proxy.

   • Misconfigured cni (Container Network Interface) plugins.

4. Configuration Errors:

   • Outdated kubelet certificates or misconfigured kubelet on the node.

   • Issues with taints and tolerations.

5. Cloud-Specific Problems:

   • Cloud provider (e.g., AWS, GCP, Azure) misconfigurations.

   • Node not attached to the cluster due to IAM or role issues.

Also don’t miss to check:

Scarcity of Resources

One common cause of the Node Not Ready error is the scarcity of resources, such as CPU or memory exhaustion. Monitoring resource usage can help identify if this is the cause. If the node is over-allocated, consider scaling down workloads or adding more nodes to the cluster.

kubelet Process

Restarting the kubelet might resolve some issues in the kubelet process. The command to restart the kubelet varies depending on the system manager in use.

sudo systemctl restart kubelet

This command restarts the kubelet service, potentially resolving issues that prevent the node from reaching a Ready state.

kube-proxy

Issues with kube-proxy, the network proxy running on each node, can also affect node readiness. Checking the status of kube-proxy and restarting it if necessary can help:

sudo systemctl status kube-proxy

This command checks the status of the kube-proxy service. If it’s not running as expected, it can be restarted with:

sudo systemctl restart kube-proxy

Restarting kube-proxy can resolve network-related issues affecting the node’s ability to communicate with the cluster, potentially resolving the Not Ready error.

Preventive Measures

We can have some preventive measures to take, so that we can safeguard our cluster to fall into the trap of this error.

• Monitor node health using tools like Prometheus or Kubernetes Dashboard.

• Set up resource limits and requests for your pods to prevent resource exhaustion.

• Regularly upgrade Kubernetes components and renew certificates.

• Automate disk cleaning or scaling using tools like Cluster Autoscaler.