Gardener: A Managed Kubernetes Service

What is Kubernetes cluster?

Source: https://kubernetes.io/docs/concepts/architecture/

A Kubernetes cluster is a set of nodes that run containerized applications. Kubernetes clusters allow containers to run across multiple machines and environments: virtual, physical, cloud-based, and on-premises. Kubernetes clusters are comprised of one master node and a number of worker nodes.

How many Kubernetes cluster we need for application?

Generally application runs on prod, preprod, staging and dev environment. We need to build separate Kubernetes cluster for each environment to provide isolation between environments.

How many Kubernetes cluster we need for entire company?

A mid size company might be running 100 or 1000 applications. This might lead to build Kubernetes clusters in the range of 1000. On top of that, company wanted to use many cloud providers (GCP, AWS, Azure etc) to keep it agnostic to provider. These leads to very high maintenance cost to build and manage Kubernetes cluster.

What is Gardener?

If you’re dealing with a lot of Kubernetes clusters (think hundreds or even thousands!) across different places like AWS, Azure, or your own data center then you can use the Gardener.

Gardener is a Kubernetes Native solution to manages Kubernetes clusters very much like pods are orchestrated in Kubernetes. Cluster workloads are scheduled and Gardenlets, similar to Kubelets, take over to manage them in particular environments in a loosely coupled, controller pattern.

It’s like a master control panel that helps you manage all these clusters easily. Gardener makes sure your clusters are always up and running, can handle lots of traffic, and are easy to update. It’s especially useful for big companies with complex setups who need a reliable and efficient way to handle their Kubernetes needs.

What is Kubeception?

Kubeception is a fun term that essentially means “Kubernetes in Kubernetes.” It describes a scenario where you’re using a Kubernetes cluster to host and manage other Kubernetes clusters.

Cluster Hierarchy in Gardener

Following is key components of Gardener.

1. Garden Cluster

• The “root” of the system. It’s a Kubernetes cluster itself, but it’s “nodeless,” meaning it only has the control plane components (API server, etcd, controller manager, scheduler).
• This cluster stores all the configuration and state for the entire Gardener landscape.
• You interact with the Garden cluster to manage all your other clusters.
• This is also called Management Cluster

2. Seed Clusters

• These are regular Kubernetes clusters that act as “hosts” for the control planes of your workload clusters (called “Shoots”).
• Each Seed cluster is typically dedicated to a specific region or cloud provider.
• They help to keep the control plane components of your workload clusters close to the worker nodes, improving performance and reducing latency.

3. Shoot Clusters:

• These are the actual Kubernetes clusters that you use to run your applications.
• Their control planes run within the Seed clusters, while their worker nodes are provisioned on the underlying infrastructure (cloud provider or on-premises).
• The Shoot resource contains the specification for the desired state of the Kubernetes cluster.
• This is also called User Cluster.

Gardener Components Overview

From a very high level point of view, the important components of Gardener are:

1. Gardener’s API Endpoint

Gardener’s API endpoint is essentially how you interact with Gardener to manage your Kubernetes clusters. It’s like the front door to the whole system.

Gardener’s API endpoint is itself a Kubernetes cluster. So you can use familiar tools like kubectl to interact with it.

Kubernetes’ API can be extended — either by CRDs or by API aggregation. Gardener adds its own custom resources (like Shoot, Seed, CloudProfile) to the API. These resources define how your clusters are configured and managed.

2. The Seeds Running the Shoot Cluster Control Planes

Inside each seed is one of the most important controllers in Gardener — the gardenlet. It spawns many other controllers, which will eventually create all resources for a shoot cluster, including all resources on the cloud providers such as virtual networks, security groups, and virtual machines.

Gardener API Resources

Since Gardener’s API endpoint is a regular Kubernetes cluster, it would theoretically serve all resources from the Kubernetes core API, including Pods, Deployments, etc. However, Gardener implements RBAC rules and disables certain controllers that make these resources inaccessible. Objects like Secrets, Namespaces, and ResourceQuotas are still available, though, as they play a vital role in Gardener.

In addition, through Gardener’s extension, the API endpoint also serves Gardener’s custom resources like Projects, Shoots, CloudProfiles, Seeds, SecretBindings (those are relevant for users), ControllerRegistrations, ControllerDeployments, BackupBuckets, BackupEntries (those are relevant to an operator), etc.

Gardener Projects

In Gardener, “Projects” are a way to group and isolate Kubernetes clusters and related resources. Under the hood, each Gardener project corresponds to a Kubernetes namespace in the Garden cluster (the management cluster).

Reconciliation in Kubernetes and Gardener

All elements of a shoot cluster have a representation in Kubernetes resources as operator and controllers are watching / acting upon them.

Based on the shoot’s specifications, Gardener will create network resources on a hyperscaler, backup resources for the ETCD, credentials, and other resources, but also representations of the worker pools. Eventually, this process will result in a fully functional shoot Kubernetes cluster.

If you change the desired state, Gardener will reconcile the shoot and run through the same cycle to ensure the actual state matches the desired state.

How to create Shoot cluster?

Following is sample shoot cluster yaml file to create user cluster on local kubernetes cluster.

# shoot-cluster.yaml
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: local
  namespace: garden-local
  annotations:
    shoot.gardener.cloud/cloud-config-execution-max-delay-seconds: "0"
    authentication.gardener.cloud/issuer: "managed"
spec:
  cloudProfile:
    name: local
  secretBindingName: local # dummy, doesn't contain any credentials
  region: local
  networking:
    type: calico
    nodes: 10.10.0.0/16
  provider:
    type: local
    workers:
    - name: local
      machine:
        type: local
      cri:
        name: containerd
      minimum: 1
      maximum: 2
      maxSurge: 1
      maxUnavailable: 0
  kubernetes:
    kubelet:
      seccompDefault: true
      serializeImagePulls: false
      registryPullQPS: 10
      registryBurst: 20
      protectKernelDefaults: true
      streamingConnectionIdleTimeout: 5m

How Does Gardener Manage the Worker Nodes?

Gardener manages the worker nodes of a Shoot using two main components: the machine-controller-manager (MCM) and the cluster-autoscaler.

The machine-controller-manager is a component that manages the lifecycle of worker nodes. It communicates with the cloud provider to create, update, or delete worker nodes as necessary. The MCM represents each worker node as a Machine resource in the control plane namespace on the Seed. It watches for changes to these Machine resources and reconciles the state of the worker nodes accordingly.

The cluster-autoscaler is an upstream component maintained by the Kubernetes community that automatically adjusts the size of the worker node pool based on the workload. If there are pods waiting for a node to be available, the cluster-autoscaler can create more worker nodes. Conversely, if there are too many idle nodes, the cluster-autoscaler can delete some worker nodes.

Shoot Cluster Hibernation and Wake-Up

In Gardener, you have the ability to put your Shoot clusters into a “hibernation” mode.

This is a cost-saving feature that allows you to remove the worker nodes in your Shoot cluster when they are not in use. When a Shoot cluster is hibernated, all the worker nodes are scaled down to zero, effectively stopping all workloads running on the cluster. Also, the control plane pods of the Shoot are scaled-down to zero replicas.

How to Manage Shoot Credentials Rotation?

Shoot clusters in Gardener require various credentials for secure communication between components. These credentials can be user-provided, such as cloud provider keys, or Gardener-provided, such as generated kubeconfig, certificate authorities, observability passwords, SSH key pair for worker nodes, ETCD encryption key, service account token signing key, etc.

Gardener uses a tool called “secrets manager” which can generate and manage all above mentioned types of secrets. Following is command to rotate the secret.

ktx garden
kubectl annotate shoot local gardener.cloud/operation=rotate-credentials-start

CloudProfile Resource

The CloudProfile resource in Gardener represents a cloud provider profile. It contains the configuration details for a specific cloud provider, such as the available machine types, regions, Kubernetes versions, and other provider-specific settings.

Project Resource

The Project resource in Gardener represents a project. Each project results in its own namespace in the Garden cluster (prefixed with garden-). A project is a logical grouping of resources, such as Shoots and Secrets.

Extension Registration

Gardener uses a concept called ControllerRegistration and ControllerDeployment to register extension controllers.

A ControllerRegistration resource represents a registration of an extension controller, while a ControllerDeployment resource contains information how to deploy the extension controller to Seed clusters. It contains the details about the extension, such as the type of resources it manages (e.g., Infrastructure or Network).

What is gardenlet?

gardenlet is a core component of Gardener that plays a crucial role in managing the control planes of Shoots. It runs on each Seed cluster and communicates with the central Gardener API server. gardenlet watches for changes to Shoot resources in the Gardener API server. When it detects a change, it reconciles the state of the corresponding control plane on the Seed.

For example, when a new Shoot is created, gardenlet creates a new namespace for the control plane on the Seed. It then creates the necessary control plane components in this namespace, such as the API server, controller manager, ETCD, and other components etc.

Each Shoot runs two ETCD clusters: events and main. These instances are separated because they store different types of data. The main ETCD instance stores most of the Kubernetes data, while the events ETCD instance stores Kubernetes events only. Separating these instances allows for better performance and isolation of the data and is a best practice for Kubernetes cluster setups.

Inspired by Kubernetes

Certainly, the architecture of Gardener is heavily inspired by the architecture of Kubernetes, which contributes to its robustness and extensibility. Just like Kubernetes, Gardener follows the control plane — worker nodes architecture.

Istio SNI in Gardener

Gardener deploys one or multiple central Istio Ingress Gateway per Seed cluster (depending on HA configuration). This gateway exposes all Kubernetes API servers of the Shoot clusters residing in that Seed. When a worker node needs to communicate with the Kubernetes API server, it sends the request to the Istio Ingress Gateway which then forwards the traffic accordingly.

Install Gardener on local

1. You can install kind with go install sigs.k8s.io/kind@v0.26.0 (for go 1.17+). This will put kind in $(go env GOPATH)/bin.
2. Run following command to clone the garnder repo

mkdir -p $(go env GOPATH)/src/github.com/gardener
cd $(go env GOPATH)/src/github.com/gardener
git clone https://github.com/gardener/gardener.git 
cd gardener

3. Run following command to to setting Up the KinD Cluster (Garden and Seed)

make kind-up

4. Run following command to Setting Up Gardener

make gardener-up

To learn more about Gardener refer https://gardener.cloud/.

Happy learning :-)