We'll begin with Kubernetes Controllers, often referred to as the brains of Kubernetes.

What is a Kubernetes Controller ?

A Kubernetes controller, in simple terms, is a component whose job is to bring the actual state of the cluster closer to its desired state. It continuously observes the resources it is responsible for and takes action whenever it detects a difference between the desired and actual states.

Kubernetes has many controllers, each responsible for a specific set of tasks. Although their responsibilities differ, they all follow the same underlying principle: reconcile the actual state with the desired state.

The Desired vs Actual state

Desired state is the state that the user or operator wants the cluster to maintain. It describes the target configuration of the cluster.

Actual state represents the current state of the cluster, based on what is actually happening at that moment. K8s components observe this state and work towards bringing it closer to the desired state.

The Working

Let us understand the working of any controller in K8s :

• Observation - The controller observe the current state of the resource it is responsible for.
• Comparison - It then compares the Desired state with the Actual state.
• Correction - if the state's don't match, it takes the necessary action. Depending on its responsibility, it might create a resource, update one, or remove one.
• Repetition - It doesn't stops after performing the task once, as changes can happen at any time, and the desired state may also change. So, it keeps reconciling.

A real controller - The ReplicaSet Controller

The ReplicaSet controller is one of the well-known controllers in Kubernetes. When we create a resource of type ReplicaSet, we specify the desired number of replicas (instances) we want to run. The job of this controller is to ensure that the number of matching Pods is equal to the desired replica count.

For example, let's say we create a ReplicaSet with replicas: 3. The controller observes the current state and finds that no matching Pods exist yet. It then requests the creation of three Pods through the API Server. Now, imagine one of these Pods gets deleted. The controller detects that only two matching Pods remain, even though the desired count is three. It requests the creation of another Pod to make up for the difference.

As discussed in the previous article, creating a Pod and running it are two different responsibilities. The ReplicaSet controller requests Pod creation, but it doesn't run the Pod itself. The Scheduler assigns the new Pod to a suitable Node, and the kubelet on that Node works with the container runtime to run its containers. We can refer to the previous article of the series for this.

This process continues as long as the ReplicaSet exists. Whether a Pod is deleted, fails, or the desired replica count changes, the controller keeps reconciling the state to bring the number of matching Pods closer to the requested count.

Other Controllers

Kubernetes has many controllers to manage different aspects of the cluster. Although their responsibilities differ, the underlying principle remains the same: to reconcile the actual state with the desired state.

For example, the Node Controller monitors the health of Nodes and responds when their state changes. Similarly, other controllers handle different responsibilities to help keep the cluster running as expected.

We'll explore these controllers in more detail as we encounter them in upcoming articles.

Thus, this marks the end of our introduction to Kubernetes Controllers!

We've seen why controllers are needed, how they work towards the desired state through reconciliation, and how the ReplicaSet controller applies this principle in practice.

But this is only the beginning. In the upcoming articles, we'll go beyond the concepts and explore how these controllers actually work under the hood by looking at the Kubernetes source code, examining debug logs, and even building a custom controller of our own.

Stay tuned for more!

You can find me here - LinkedIn Instagram Twitter(X)

Kubernetes components generally don't communicate directly with one another. Instead, they interact through the Kubernetes API Server, which acts as a common point of communication. This allows components to focus on their own responsibilities without needing to know the internal workings of other components.

Components make read and write requests to the API Server to retrieve or update resources. But how do they know when something changes? One approach would be to keep sending requests to the API Server at regular intervals to check for updates. However, this approach has a few disadvantages:

• Unnecessary requests: Components may keep making requests even when nothing has changed.
• Delayed updates: A component may not detect a change until its next scheduled check.
• Increased load: Frequent polling by multiple components can generate a large number of requests to the API Server.

This is where the Kubernetes Watch Mechanism comes into the picture. Instead of repeatedly requesting the latest state, components can watch for changes to the resources they care about and receive notifications when relevant changes occur.

The Working of the Mechanism

In K8s, the Watch mechanism uses long-lived HTTP connections to observe changes in resources. It is an event-driven mechanism mediated by the API Server. Components establish a connection with the API Server through the Kubernetes Watch API and listen for relevant changes to the resources they are interested in.

Long lived HTTP connection

A long-lived HTTP connection is not a separate protocol. It is a connection that remains open for an extended period, and there are different ways to use it. Two examples are HTTP streaming and Server-Sent Events (SSE).

In these approaches, the client sends an HTTP request, and the server keeps the response open to stream data back to the client. However, the client generally cannot send arbitrary new application messages through that same response. To send another request, it uses a separate HTTP request.

This differs from WebSocket, which support full-duplex communication, meaning both the client and server can send data to each other over the same connection.

Let us go step by step to understand the working for a component :

• Initial Request (LIST) - The component first performs a LIST operation.It asks the API Server to return a list of resources of a particular type. The response looks something like this:

  // API call : GET /api/v1/pods
  {
      "kind": "PodList",
      "apiVersion": "v1",
      "metadata": {
          "resourceVersion": "12345",
          ...others
       },
      "items": [
          ....
      ]
  }
  

  Themetadata.resourceVersionfield identifies the version associated with the returned snapshot. The component keeps track of this version so it can use it when starting the Watch request.

• Watch Request - Now, the component opens a watch request with the API Server GET /api/v1/pods?watch=true&resourceVersion=12345. This establishes a long-lived HTTP connection between the component and the API Server. The server keeps the response open and streams relevant changes that occurred after the specified resource version.

• Streaming Updates - Whenever a relevant change occurs, such as a Pod being created, updated, or deleted, the API Server sends an event over the same connection. For example:

  {
    "type": "ADDED",
    "object": { ...... }
  }
  
  {
    "type": "MODIFIED",
    "object": { ...... }
  }
  
  {
    "type": "DELETED",
    "object": { ...... }
  }
  

  The component processes these events and takes the appropriate action. For example, when the Scheduler learns about a new Pod that has not yet been assigned to a Node, it can evaluate that Pod for scheduling.

However, this connection does not remain open indefinitely. It may close because of a timeout, a network interruption, or other reasons. The component must handle these situations and reconnect when needed.

To resume watching, it can use the latest resourceVersion it has processed, allowing it to receive subsequent changes as long as that version is still available in the API Server's history. If the requested version is no longer available, the API Server can return a 410 Gone response. The component must then perform another LIST operation to obtain a fresh snapshot and start watching again using the new version.

An Example

Let us understand this complete interaction with the help of a simple example - Creating a Pod. We will assume that the Scheduler already have an established connection.

1. Create a Pod - We send a request to the API Server using kubectl to create a Pod. After validation, the K8s stores the Pod's data in etcd.
2. API Server sends an event - Since the Scheduler have a connection established, so the API Server streams an event with the new Pod's details over the same connection.
3. Scheduler starts focusing on the Pod - The Scheduler then decides the suitable node for the POD based on Pod's requirement and scheduling constraints.
4. Updating the API Server - Once the node is assigned to the Pod, the Scheduler makes a separate API request to the update the information in the etcd.
5. Running the Pod - On the assigned Node, the kubelet got the Pod's specs through its own watcher with the API Server. It then works on that Node to run the Pod.

Illustration

This design let's Kubernetes to scale to thousands of nodes — components don’t spam the API Server, they just “listen” for updates.

In the upcoming articles in this series, we'll go more deeper into the Kubernetes world.

Stay tuned for more!

You can find me here - LinkedIn Instagram Twitter(X)

In today's AI-driven world, where we rely on AI to get things done faster, it's easy to overlook the fundamentals of the systems we use every day.

From my experience, I believe it's important to understand how these systems work under the hood. When things break, services go down, or a blackout hits, knowing the internals helps you troubleshoot problems, make informed decisions, and get things back on track.

AI can help us build faster, but understanding the fundamentals helps us when things go wrong.

Kubernetes or k8s, in simple terms, is a system that helps us deploy, manage, and scale applications. Originally developed by Google, it automates many of the tasks involved in running applications, making them easier to manage reliably at scale.

To understand Kubernetes and what it does, we first need to understand its resources and components. It wouldn't be entirely incorrect to say that these components, working together, are what make Kubernetes what it is.

There are, however, many resources and components in Kubernetes. As the title of this blog suggests, we'll keep things simple and won't cover every one of them here. Instead, we'll follow a learn as you go approach.

This approach has personally worked well for me, as too much theoretical information without practical context can make it difficult to understand how things actually work.

Before we dive in the k8s components role's and their working, let us learn some essential terms to start with.

• Pod - The smallest unit in K8s. It can contain one or more containers but the application container runs here.

• Node - A physical or virtual machine where application workloads run . It has its own CPU, memory, and storage.

• Cluster - A group of machine(s), or node(s), managed by Kubernetes to deploy and manage applications.

Now, let's take a look at some of the key components of Kubernetes and understand their roles. These components form the backbone of K8s. These components are each responsible for a different task. To understand them better, let's group them based on their roles and where they typically run.

Control Plane components

These components are responsible for managing the cluster and making decisions about its state.

• API Server - This is the main entry point for interacting with the Kubernetes API. It validates, stores and exposes K8s resources. We can think of it as the gateway through which users and Kubernetes components interact with the cluster.

• etcd - This is the Kubernetes database. It is a key-value store that holds the cluster's data, including the desired and current states recorded in Kubernetes resources.

• Scheduler -This is the Pod assigner. Its primary job is to assign a Pod to a Node based on multiple factors, such as node availability, the Pod's requirements, and other scheduling constraints.

• Controllers - These are the brains of K8s. They make sure what should be is what is. They continuously monitor resources and take action to bring the actual state of the cluster closer to the desired state.

Worker Node Components

These components help run and manage application workloads on nodes.

• kubelet — This is a node-level agent that runs on every worker node (and sometimes on control plane nodes too). It is responsible for ensuring that the Pods assigned to its node are running as expected.

Before wrapping up our discussion of components, let's look at one more tool: kubectl (pronounced “cube control”). It is a CLI tool that we, as users or administrators, use to communicate with the API Server.

These are just a few high-level components that will help us understand the lifecycle of an application in Kubernetes. K8s has many other components, which we will explore in later parts of this series as and when we use them.

Now, it's time to bring together the components we've discussed above and see how they work together when we create a resource in K8s -

Whenever we create a resource using kubectl, it sends a request to the API Server. The API Server validates the request and stores the resource's data in etcd. After that, different components act depending on the type of resource we created.

For a Pod that hasn't been assigned to a Node yet, the Scheduler picks it up and selects a suitable Node based on factors such as resource availability and the Pod's requirements. Meanwhile, Controllers continuously monitor resources and take action to bring the cluster's actual state closer to its desired state.

Once a Node is assigned to the Pod, the kubelet on that Node takes over. It works with the container runtime to ensure the Pod's containers run as expected.

Now, comes the best part :-

A Real world analogy - The City of 100 Streetlight

Let us imagine Kubernetes as a system that needs to maintain 100 streetlights across the city 🌆

Component	Role in the City
kubectl	This can be the city planner submitting a proposal "Install 100 streetlights."
API Server	The city’s central office that records every plan and status update. It doesn’t build anything, just coordinates communication.
etcd	The city’s permanent record book — stores both plans and current statuses.
Scheduler	The city’s logistics officer: “Put these 10 lights on Street A, 20 on Street B…” (decides placement)
Controller	The maintenance supervisor: “One light is broken, replace it” or “We need 100 total, only 98 exist, add 2 more.
kubelet	The electrician on each street — actually installs, powers, and maintains the lights assigned to that street. Reports back when lights are on or off.

With this analogy, we can related how all these components stich together and help us in achieving the goal.

With this, we come to the end of this article!

K8s is much, much more than what we've discussed above, but these concepts can serve as a great starting point for beginners or a quick refresher for those revisiting the basics.

In the upcoming articles in this series, we'll dive deeper into Kubernetes components and their inner workings, with practical examples and animations to make things easier to understand.

Stay tuned for more!

You can find me here - LinkedIn Instagram Twitter(X)