SAP Gardener architecture

10 min readFeb 4, 2025

1.0 Introduction to Gardener

Gardener is an open-source system for managing Kubernetes clusters. It allows you to provision and manage clusters on various cloud providers and on-premises infrastructure in a consistent and scalable way.

1.1 The Gardener Landscape

Gardener introduces a hierarchy of Kubernetes clusters:

Garden Cluster: The root cluster, the “control plane of control planes.” It hosts the core Gardener components responsible for managing all other clusters.
Seed Clusters: Intermediate clusters that act as templates or blueprints for creating Shoot clusters. They provide a layer of isolation and allow for customization based on specific requirements.
Shoot Clusters: The actual Kubernetes clusters where your applications run. These are the “end-user” clusters that developers interact with.

1.2 Key Concepts

Kubeception: Gardener uses Kubernetes to manage Kubernetes clusters. This allows it to leverage Kubernetes’ existing features and capabilities for resource management, scheduling, and scaling.
Control Plane: Each cluster (Garden, Seed, and Shoot) has its own control plane, which consists of components like the API server, controller manager, and etcd.
Worker Nodes: These are the machines that run your applications in the Shoot clusters.
Extensions: Gardener supports extensions that allow you to customize and extend its functionality. For example, you can use extensions to integrate with different cloud providers or add support for specific tools and services.

Chapter 2: Garden Cluster Architecture

The Garden cluster is the foundation of your Gardener environment, acting as the central control plane for managing all other clusters (Seed and Shoot) and ensuring the smooth operation of the entire system. It’s the “brain” or “command center” of your Gardener landscape.

2.1 Key Components and Responsibilities

The Garden cluster consists of several essential components, each playing a crucial role in managing and orchestrating the Gardener environment. These components work together to provide a robust and scalable platform for managing Kubernetes clusters.

Gardener API Server: The central API server for interacting with Gardener. You use this API server to manage Seed clusters, provision Shoot clusters, and monitor the overall system.

2. Controller Manager: The heart of the control plane. It runs various controllers that handle different aspects of Gardener’s functionality, such as:

Seed Management: Managing the lifecycle of Seed clusters, ensuring they are healthy and properly configured.
Shoot Provisioning: Provisioning new Shoot clusters based on user requests and managing their lifecycle.
Networking: Managing network connectivity between clusters and components, including VPC peering and VPNs.
Extensions: Managing the lifecycle of Gardener extensions, which provide additional functionality and integrations.

3. Scheduler: Responsible for scheduling Shoot control planes onto Seed clusters. It considers factors like resource availability, constraints, and policies when making scheduling decisions.

4. etcd: The key-value store for Gardener’s configuration and state data. It ensures the consistency and reliability of the Gardener system by providing a persistent storage mechanism for critical information.

5. Gardener Operator: Manages the lifecycle of Gardener components, ensuring that they are properly deployed and configured. It automates tasks such as deploying updates, handling failures, and scaling components.

6. Webhook Server: Acts as a gatekeeper for Kubernetes resources. It validates and mutates resources before they are persisted in etcd, ensuring that they comply with Gardener’s policies and security requirements.

7. Storage: Provides persistent storage for various Gardener components, such as etcd and the Gardener API server. This could be a cloud-managed storage service or an on-premises storage solution.

8. Garden Cluster API LB: A load balancer that distributes incoming traffic to the Gardener API server, ensuring high availability and scalability.

9. Ingress LB: A load balancer that distributes incoming traffic to various Gardener services, such as the Gardener dashboard and other internal services.

10. Workers: Worker nodes in the Garden cluster run various workloads, including the Gardener controllers, the Gardener Operator, and other supporting services.

11. Kubelet + Container Runtime: Each worker node runs a Kubelet and a container runtime (such as Docker or containerd) to manage containers and pods.

2.2 User Interfaces

Gardener Dashboard: A web-based user interface for interacting with the Gardener API server. It provides a visual representation of your Gardener landscape, allowing you to manage clusters, monitor resources, and troubleshoot issues.
gardenctl CLI: A command-line interface for interacting with the Gardener API server. It provides a more programmatic way to manage Gardener resources and automate tasks.

# Using gardenctl vs kubectl
## Get seed
$ gardenctl get seed my-seed
$ kubectl get seed my-seed -n garden
## Creating a Shoot Cluster
$ gardenctl create shoot my-shoot --seed my-seed --secret my-cloud-credentials --kubeconfig my-seed-kubeconfig
$ kubectl apply -f shoot.yaml
## Listing Shoot Clusters
$ gardenctl list shoots
$ kubectl get shoots --all-namespaces
## Getting Shoot Cluster Details
$ gardenctl get shoot my-shoot
$ kubectl describe shoot my-shoot -n my-project
## Deleting a Shoot Cluster
$ gardenctl delete shoot my-shoot
$ kubectl delete shoot my-shoot -n my-project
## Managing Deployments (within a Shoot)
$ gardenctl does not have direct commands for managing deployments within a Shoot.
$ kubectl scale deployment my-app --replicas=3 -n my-namespace

2.3 External Access and Integration

Integration with Cloud Providers: Gardener integrates with various cloud providers (AWS, Azure, GCP, etc.) to provision and manage infrastructure resources for Seed and Shoot clusters.
Integration with Monitoring and Logging Systems: Gardener can be integrated with external monitoring and logging systems to provide observability into the health and performance of the Gardener environment.

Chapter 3: Seed Cluster Architecture

Seed clusters in Gardener are intermediate clusters that serve as the foundation for creating and managing Shoot clusters. They act as templates or blueprints, allowing you to customize the environment and configuration for your Shoot clusters based on specific requirements such as region, cloud provider, or machine types.

3.1 Key Components and Responsibilities

Seed API Server: The central point of interaction for managing the Seed cluster itself. It exposes the Kubernetes API for managing resources within the Seed cluster.

2. Seed Controller Manager: Runs various controllers that are responsible for managing different aspects of the Seed cluster’s functionality. These controllers handle tasks such as:

Managing the lifecycle of Shoot clusters within the Seed.
Monitoring the health and status of Shoot clusters.
Scaling Shoot clusters based on demand.
Updating Shoot clusters with new configurations or Kubernetes versions.

3. Seed etcd: The key-value store for the Seed cluster’s configuration and state data. It ensures the consistency and reliability of the Seed cluster by providing a persistent storage mechanism for critical information.

4. Gardenlet: An agent that runs in each Seed cluster and communicates with the Garden cluster. It acts as the “local gardener” for the Seed cluster, responsible for:

Receiving instructions from the Garden cluster to create, update, or delete Shoot clusters.
Managing the lifecycle of Shoot control planes within the Seed cluster.
Monitoring the health and status of Shoot clusters and reporting back to the Garden cluster.

5. Kubelet + Container Runtime: Each worker node in the Seed cluster runs a Kubelet and a container runtime (such as Docker or containerd) to manage containers and pods. This is essential for running the various controllers and agents within the Seed cluster.

6. ETCD Main: The main etcd cluster for the Seed cluster, storing critical configuration and state data.

7. ETCD Events: A separate etcd cluster used specifically for storing events related to the Seed cluster and its Shoot clusters.

8. VPN Server: Establishes and manages VPN connections between the Seed cluster and the Shoot clusters it manages. This provides secure communication channels for managing and monitoring the Shoots.

9. Envoy: A sidecar proxy used for various networking tasks within the Seed cluster, such as traffic routing and management.

10. Machine Controller: Manages the lifecycle of worker nodes in the Shoot clusters, including provisioning, scaling, and updates.

11. Monitoring: Components responsible for monitoring the health and performance of the Seed cluster and its Shoot clusters. This typically includes metrics collection and alerting.

12. Logging: Components responsible for collecting and storing logs from various components within the Seed cluster and its Shoot clusters.

13. Storage: Provides persistent storage for various Seed cluster components, such as etcd and other stateful applications.

14. Seed Cluster API LB: A load balancer that distributes incoming traffic to the Seed API server, ensuring high availability and scalability.

15. Istio Ingress Gateway LB: A load balancer that distributes incoming traffic to the Istio Ingress Gateway, which manages external access to services within the Seed cluster.

3.2 Shoot Control Plane

The Shoot control plane runs within the Seed cluster, providing a secure and isolated environment for managing the Shoot clusters. This includes components such as:

Shoot API Server: The API server for a specific Shoot cluster, used to manage resources and interact with the Shoot cluster.
Shoot Controller Manager: Runs controllers responsible for managing the Shoot’s worker nodes and other components.
Shoot etcd: The key-value store for the Shoot cluster’s configuration and state.

A Shoot control plane manages only one Shoot cluster

Each Shoot cluster has its own dedicated control plane. This control plane is responsible for managing the resources and worker nodes within that specific Shoot cluster. It doesn’t manage any other Shoot clusters.

Multiple Shoot Control Planes per Seed

A single Seed cluster can have multiple Shoot control planes running within it. Here’s why and how it works:

Seed as a Host: Think of a Seed cluster as a hosting environment for Shoot control planes. It provides the necessary resources (compute, storage, networking) for these control planes to operate.
Isolation: Each Shoot control plane, even though running within the same Seed, is isolated in its own namespace and has its own dedicated resources (etcd, API server, etc.). This ensures that one Shoot control plane doesn’t interfere with others.
Scalability: A single Seed cluster can typically host multiple Shoot control planes, allowing Gardener to manage a large number of Shoot clusters efficiently. The actual number of Shoot control planes a Seed can handle depends on factors like the Seed’s resource capacity and configuration.
Management: The gardenlet agent running in the Seed cluster is responsible for managing the lifecycle of these Shoot control planes, ensuring they are healthy and properly configured.

Benefits of Multiple Shoot Control Planes per Seed

Resource Efficiency: Reduces resource consumption by consolidating multiple Shoot control planes within a single Seed cluster.
Simplified Management: Makes it easier to manage and monitor multiple Shoot clusters from a central Seed cluster.
Cost Optimization: Can potentially reduce costs by sharing infrastructure resources among multiple Shoot control planes.

3.3 Networking

Connectivity with Garden: Seed clusters maintain secure communication channels with the Garden cluster for exchanging information and receiving instructions.
Connectivity with Shoots: Seed clusters provide network connectivity to the Shoot clusters they manage, often through VPC peering or VPNs.

Chapter 4: Shoot Cluster Architecture

Shoot clusters are the stars of the Gardener show. They are the actual Kubernetes clusters where your applications run, where developers deploy their workloads, and where the magic of container orchestration happens. Gardener provides a robust and flexible framework for managing these Shoot clusters, ensuring they are secure, scalable, and highly available.

4.1 Key Components and Responsibilities

Worker Nodes: The workhorses of your Shoot cluster. These are the VMs or physical machines where your applications and services are deployed and run. Each worker node runs a Kubelet and a container runtime (such as Docker or containerd) to manage containers and pods.

2. Kubelet: The agent that runs on each worker node. It’s responsible for:

Communicating with the Shoot API server to receive instructions and report node status.
Managing containers and pods on the node, ensuring they are running and healthy.
Mounting volumes and providing network connectivity to pods.

3. (Reverse) VPN Client: Establishes a secure VPN connection back to the Seed cluster. This allows the Shoot cluster to be managed and monitored from the Seed cluster, even if it’s deployed in a different network or cloud provider.

4. Calico/Cilium: The Container Network Interface (CNI) responsible for providing networking for pods and services within the Shoot cluster. It handles tasks such as IP address allocation, routing, and network policy enforcement.

5. Kube Proxy: Runs on each worker node and is responsible for maintaining network rules and routing traffic to pods and services.

6. CoreDNS: Provides DNS resolution for services within the Shoot cluster, allowing applications to discover and communicate with each other using service names.

4.2 Optional Add-ons

Shoot clusters can include various optional add-ons to provide additional functionality, such as:

Monitoring: Tools and services for monitoring the health and performance of applications and the cluster itself. This might include Prometheus, Grafana, and other monitoring agents.

2. Logging: Tools and services for collecting and storing logs from applications and cluster components. This might include Fluentd, Fluent Bit, and Elasticsearch.

3. Service Mesh: A service mesh like Istio can be deployed to provide advanced traffic management, security, and observability for microservices-based applications.

4. Ingress Controller: An ingress controller like Nginx Ingress or Istio Ingress Gateway can be used to manage external access to applications running in the Shoot cluster.

4.3 Actual Workload

This is where your applications and services reside. You deploy your workloads (Deployments, StatefulSets, DaemonSets, etc.) onto the worker nodes in the Shoot cluster, and they run within containers managed by the Kubelet and container runtime.

Chapter 4: Detailed architecture diagram

This post is based on https://gardener.cloud/docs/gardener/concepts/architecture/ and interaction with https://gemini.google.com/.

Happy learning :-)