Gardner cluster IPAddress and DNS management
Chapter 1: Introduction to IP Management in Gardener
What is IP Management (IPAM)?
IPAM (IP Address Management) is the process of planning, tracking, and managing the allocation and usage of IP addresses in a network.
Importance in Gardener?
- Ensures efficient and conflict-free allocation of IP addresses for resources in
Shoot,Seed, andGardenclusters. - Supports dynamic scaling and automation in Kubernetes clusters.
Chapter 2: IP Management in Shoot Clusters
2.1 Overview
- Shoot clusters are end-user Kubernetes clusters managed by Gardener.
- IP management is critical for nodes, Pods, services, and load balancers.
2.2 Node IP Allocation
Worker nodes are assigned IP addresses from the node CIDR specified in the Shoot resource.
The infrastructure controller creates a subnet in the cloud provider’s VPC. The Machine Controller Manager (MCM) provisions worker nodes, and the cloud provider assigns IPs from the subnet.
Sample Shoot Configuration for Nodes IP Allocation
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: my-shoot
namespace: garden-my-project
spec:
networking:
nodes: 10.250.0.0/16 # CIDR for Nodes
provider:
type: gcp
infrastructureConfig:
networks:
workers: 10.250.0.0/16 # Subnet for worker nodesgcp extension controller code reference
The infrastructure controller creates subnets in the GCP VPC:
// pkg/controller/infrastructure/reconcile.go
func (a *actuator) reconcileInfrastructure(ctx context.Context, infra *extensionsv1alpha1.Infrastructure, cluster *controller.Cluster) error {
client, err := a.getGCPClient(ctx, infra, cluster)
if err != nil {
return err
}
// Create or update the subnet
subnet, err := client.CreateOrUpdateSubnet(ctx, infra, cluster)
if err != nil {
return err
}
// Update the infrastructure status
infra.Status.ProviderStatus = &runtime.RawExtension{
Object: &gcpapi.InfrastructureStatus{
Subnets: []gcpapi.Subnet{
{
Name: subnet.Name,
CIDR: subnet.CIDR,
},
},
},
}
return nil
}The infrastructure controller configures firewall rules:
// pkg/controller/infrastructure/reconcile.go
func (a *actuator) reconcileFirewallRules(ctx context.Context, client gcpclient.Interface, infra *extensionsv1alpha1.Infrastructure, cluster *controller.Cluster) error {
rules := []gcpapi.FirewallRule{
{
Name: "allow-ssh",
Network: cluster.Shoot.Spec.Networking.Nodes,
SourceRanges: []string{"0.0.0.0/0"},
Allowed: []gcpapi.Allowed{
{
Protocol: "tcp",
Ports: []string{"22"},
},
},
},
}
for _, rule := range rules {
if err := client.CreateOrUpdateFirewallRule(ctx, rule); err != nil {
return err
}
}
return nil
}2.3 Pod IP Allocation
Pods are assigned IP addresses from the pod CIDR specified in the Shoot resource. The CNI plugin (e.g., Calico, Cilium) manages IP allocation for Pods.
Sample Shoot Configuration for Pod IP Allocation
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: my-shoot
namespace: garden-my-project
spec:
networking:
pods: 100.96.0.0/11 # CIDR for Pods2.4 Service IP Allocation
Kubernetes services (e.g., ClusterIP, LoadBalancer) are assigned IP addresses from the service CIDR.
For LoadBalancer services, the cloud provider allocates a public IP and configures a load balancer.
Sample Shoot Configuration for Service IP Allocation
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: my-shoot
namespace: garden-my-project
spec:
networking:
services: 100.64.0.0/13 # CIDR for Services2.5 API Server IP Allocation
The Kubernetes API server requires a public or private IP address.
The controlplane controller allocates an IP for the API server.
Sample Shoot Configuration for Service IP Allocation
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
spec:
provider:
type: gcp
controlPlaneConfig:
apiServer:
dns:
domain: api.example.com
service:
type: LoadBalancer
loadBalancerIP: 203.0.113.10 # Optional: Static IP for API serverChapter 3: IP Management in Seed Clusters
3.1 Overview
- Seed clusters host the control plane of Shoot clusters.
- IP management is required for control plane components and networking.
3.2 Control Plane IPs
Control plane components (e.g., API server, etcd) need IP addresses. The Seed controller manages IP allocation for control plane components.
3.3 LoadBalancer IPs
If the API server is exposed via a load balancer, a public IP address is required.
The cloud provider allocates a public IP for the load balancer.
Chapter 4: IP Management in Garden Clusters
4.1 Overview
- The Garden cluster is the management cluster for Gardener.
- IP management is required for the Gardener control plane.
4.2 Control Plane IPs
Gardener control plane components (e.g., Gardener API server) need IP addresses.
IP management is typically handled manually or using external tools (e.g., Terraform).
Chapter 5: IP Allocation using AddressPoolClaim
5.1 AddressPoolClaim
A custom resource for requesting IP addresses from a predefined pool. For example, allocating static IPs for LoadBalancer services or API servers.
Sample AddressPoolClaim
apiVersion: networking.gardener.cloud/v1alpha1
kind: AddressPoolClaim
metadata:
name: my-address-claim
spec:
poolRef:
name: my-address-pool
purpose: LoadBalancer
status:
allocatedAddress: 203.0.113.105.2 SubnetClaim
A custom resource for requesting subnets. It used to allocate subnets for worker nodes.
Sample SubnetClaim:
apiVersion: networking.gardener.cloud/v1alpha1
kind: SubnetClaim
metadata:
name: worker-node-network-subnet
spec:
cidr: 10.250.0.0/16Chapter 6: Introduction to DNS Management in Gardener
6.1 What is DNS Management?
DNS (Domain Name System) management involves configuring and managing domain names and their mappings to IP addresses.
Importance in Gardener
- Enables human-readable domain names for accessing Kubernetes clusters and services.
- Supports dynamic DNS record creation and updates for Shoot clusters.
Chapter 7: DNS Management in Shoot Clusters
7.1 Overview
Shoot clusters require DNS management for:
- API Server: A domain name for accessing the Kubernetes API server.
- Ingress: Domain names for applications exposed via ingress controllers.
- Services: Domain names for services of type
LoadBalancer.
7.2 DNSRecord Resource
A custom resource (DNSRecord) is used to define DNS records for Shoot clusters.
Purpose: Maps domain names to IP addresses (e.g., for the API server or ingress).
Sample DNSRecord CRD
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSRecord
metadata:
name: api-server-dns-record
namespace: garden-my-project
spec:
zone: example.com.
name: api.example.com.
recordType: A
values:
- 203.0.113.10
ttl: 300
status:
observedGeneration: 1
state: Ready
message: "DNS record successfully reconciled"DNSRecord Reconciliation
The DNS controller reconciles DNSRecord resources:
// pkg/controller/dnsrecord/actuator.go
func (a *actuator) Reconcile(ctx context.Context, dns *extensionsv1alpha1.DNSRecord, cluster *extensionscontroller.Cluster) error {
// Get the DNS provider client
client, err := a.getDNSClient(ctx, dns, cluster)
if err != nil {
return err
}
// Create or update the DNS record
if err := client.CreateOrUpdateDNSRecord(dns.Spec.Zone, dns.Spec.Name, dns.Spec.RecordType, dns.Spec.Values, dns.Spec.TTL); err != nil {
return err
}
// Update the DNSRecord status
dns.Status.ObservedGeneration = dns.Generation
dns.Status.State = extensionsv1alpha1.DNSRecordStateReady
dns.Status.Message = "DNS record successfully reconciled"
return nil
}7.3 DNSProvider Resource
A custom resource (DNSProvider) defines the DNS provider (e.g., GCP Cloud DNS, AWS Route 53) and credentials.
Purpose: Specifies where DNS records should be created.
Sample DNSProvider CRD:
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSProvider
metadata:
name: gcp-dns-provider
namespace: garden-my-project
spec:
type: google-clouddns
secretRef:
name: gcp-dns-credentials
domains:
include:
- example.com.DNSProvider Client
The DNS controller interacts with the DNS provider:
// pkg/controller/dnsrecord/client.go
type DNSClient interface {
CreateOrUpdateDNSRecord(zone, name, recordType string, values []string, ttl int) error
DeleteDNSRecord(zone, name, recordType string) error
}
type GCPDNSClient struct {
client *dns.Service
}
func (c *GCPDNSClient) CreateOrUpdateDNSRecord(zone, name, recordType string, values []string, ttl int) error {
// Logic to create or update DNS record in GCP Cloud DNS
return nil
}
func (c *GCPDNSClient) DeleteDNSRecord(zone, name, recordType string) error {
// Logic to delete DNS record in GCP Cloud DNS
return nil
}7.4 DNS Controller
The DNS controller reconciles DNSRecord resources and interacts with the DNS provider to create/update DNS records.
Chapter 8: DNS Management in Seed Clusters
8.1 Overview
Seed clusters host the control plane of Shoot clusters. DNS management is required for:
- Control Plane: Domain names for the API server and other control plane components.
- Networking: Domain names for internal communication.
8.2 Control Plane DNS
The API server and other control plane components need domain names for external access.
The Seed controller manages DNS records for control plane components.
8.3 Internal DNS
Domain names for internal communication between Seed and Shoot clusters.
The Seed controller configures internal DNS records.
Chapter 9: DNS Management in Garden Clusters
9.1 Overview
The Garden cluster is the management cluster for Gardener. DNS management is required for:
- Gardener Control Plane: Domain names for the Gardener API server and other components.
- Networking: Domain names for internal communication.
9.2 Gardener Control Plane DNS
The Gardener API server and other components need domain names for external access.
DNS management is typically handled manually or using external tools (e.g., Terraform).
Chapter 10: DNS Management
10.1 DNS CNAME Records
CNAME records are used to alias one domain name to another.
Use Case: Mapping a custom domain name to the API server or ingress.
Sample DNSRecord for CNAME:
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSRecord
metadata:
name: app-cname-record
namespace: garden-my-project
spec:
zone: example.com.
name: app.example.com.
recordType: CNAME
values:
- example.com.
ttl: 30010.2 DNS TXT Records
TXT records are used to store text data, often for verification or configuration.
Use Case: Domain verification for TLS certificates.
Sample DNSRecord for TXT
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSRecord
metadata:
name: verification-txt-record
namespace: garden-my-project
spec:
zone: example.com.
name: example.com.
recordType: TXT
values:
- "v=spf1 include:_spf.example.com ~all"
ttl: 300Chapter 11: Subnet, IPAddress and DNS management flow
11.1 Overview
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Shoot Resource | | Infrastructure | | GCP Cloud DNS |
| (CIDR Ranges) | | Controller | | (DNS Records) |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| 1. Define CIDR Ranges | |
+-------------------------->| |
| | 2. Create Subnet in VPC |
| +-------------------------->|
| | |
| | 3. Allocate Subnet |
| +-------------------------->|
| | |
| | 4. Provision Worker Nodes |
| +-------------------------->|
| | |
| | 5. Assign IPs to Nodes |
| +-------------------------->|
| | |
| | 6. Create DNSRecord CR |
| +-------------------------->|
| | |
| | 7. Map IP to DNS Record |
| +-------------------------->|
| | |
| | 8. Update DNS Provider |
| +-------------------------->|
| | |
+--------+----------+ +--------+----------+ +--------+----------+
| | | | | |
| Node IPs | | DNS Controller | | DNS Records |
| (e.g., 10.250.0.2)| | | | (e.g., api.example.com)|
| | | | | |
+-------------------+ +-------------------+ +-------------------+11.2 Step-by-Step Flow
Step 1: Define CIDR Ranges in Shoot Resource
The user specifies the node CIDR, pod CIDR, and service CIDR in the Shoot resource.
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: my-shoot
namespace: garden-my-project
spec:
networking:
nodes: 10.250.0.0/16 # CIDR for Nodes
pods: 100.96.0.0/11 # CIDR for Pods
services: 100.64.0.0/13 # CIDR for Services
provider:
type: gcp
infrastructureConfig:
networks:
workers: 10.250.0.0/16 # Subnet for worker nodesStep 2: Infrastructure Controller Creates Subnet
The Infrastructure Controller creates a subnet in the GCP VPC based on the node CIDR specified in the Shoot resource.
// pkg/controller/infrastructure/reconcile.go
func (a *actuator) reconcileInfrastructure(ctx context.Context, infra *extensionsv1alpha1.Infrastructure, cluster *controller.Cluster) error {
client, err := a.getGCPClient(ctx, infra, cluster)
if err != nil {
return err
}
// Create or update the subnet
subnet, err := client.CreateOrUpdateSubnet(ctx, infra, cluster)
if err != nil {
return err
}
// Update the infrastructure status
infra.Status.ProviderStatus = &runtime.RawExtension{
Object: &gcpapi.InfrastructureStatus{
Subnets: []gcpapi.Subnet{
{
Name: subnet.Name,
CIDR: subnet.CIDR,
},
},
},
}
return nil
}Step 3: Machine Controller Manager (MCM) Provisions Worker Nodes
- The Machine Controller Manager (MCM) provisions worker nodes in the specified subnet.
- GCP assigns IP addresses to the worker nodes from the subnet (
10.250.0.0/16).
Step 4: Assign IPs to Nodes
Each worker node is assigned an IP address from the subnet (e.g., 10.250.0.2, 10.250.0.3).
Step 5: Create DNSRecord Resource
The Shoot Controller in the Gardener Control Plane is responsible for creating the DNSRecord resource to map the API server’s IP address to a domain name (e.g., api.example.com).
The Shoot Controller reads the Shoot resource’s configuration to determine the domain name and IP address for the API server.
The domain name is typically derived from the Shoot resource’s spec.dns.domain field.
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: my-shoot
namespace: garden-my-project
spec:
dns:
domain: example.com # Base domain for the Shoot cluster
networking:
nodes: 10.250.0.0/16
provider:
type: gcp
controlPlaneConfig:
apiServer:
service:
type: LoadBalancer
loadBalancerIP: 203.0.113.10 # Optional: Static IP for API serverOnce the DNSRecord resource is created, the DNS Controller takes over to reconcile the resource and create the actual DNS record in the DNS provider (e.g., GCP Cloud DNS).
Step 6: DNS Controller Reconciles DNSRecord
The DNS Controller watches for DNSRecord resources. When a DNSRecord is created or updated, the DNS Controller:
- Reads the
DNSRecordspecification (e.g., domain name, IP address, record type). - Interacts with the DNS provider (e.g., GCP Cloud DNS) to create or update the DNS record.
- Updates the
DNSRecordstatus to reflect the reconciliation result.
// pkg/controller/dnsrecord/actuator.go
func (a *actuator) Reconcile(ctx context.Context, dns *extensionsv1alpha1.DNSRecord, cluster *extensionscontroller.Cluster) error {
client, err := a.getDNSClient(ctx, dns, cluster)
if err != nil {
return err
}
// Create or update the DNS record
if err := client.CreateOrUpdateDNSRecord(dns.Spec.Zone, dns.Spec.Name, dns.Spec.RecordType, dns.Spec.Values, dns.Spec.TTL); err != nil {
return err
}
// Update the DNSRecord status
dns.Status.ObservedGeneration = dns.Generation
dns.Status.State = extensionsv1alpha1.DNSRecordStateReady
dns.Status.Message = "DNS record successfully reconciled"
return nil
}Step 7: Map IP to DNS Record
The DNS provider (e.g., GCP Cloud DNS) creates a DNS record mapping the domain name (api.example.com) to the IP address (203.0.113.10).
Step 8: Update DNS Provider
The DNS provider updates its records, making the domain name (api.example.com) resolvable to the IP address (203.0.113.10).
This post is based on interaction with https://chat.deepseek.com/. LLM is changing the way we study and write :)
Enjoy learning Gardener’s internals :-)
