SAP etcd operator etcd-druid

Chapter 1: Introduction to etcd and the Need for etcd-druid

1.1 What is etcd?

1.1.1 Key-Value Store Fundamentals

etcd is a strongly consistent, distributed key-value store. Let’s break down what that means:

• Key-Value Store: Data is stored in the form of key-value pairs. Think of it like a dictionary or hash map. A key is a unique identifier (usually a string, like /config/database/url), and a value is the data associated with that key (e.g., postgres://user:password@host:port/database). Keys are often organized hierarchically, like a filesystem, which makes it intuitive to represent configuration data.
• Distributed: etcd is designed to run across multiple machines (a cluster) rather than on a single server. This provides fault tolerance and high availability. If one machine fails, the others can continue to operate.
• Strongly Consistent: This is a crucial aspect. etcd uses the Raft consensus algorithm to ensure that all nodes in the cluster agree on the current state of the data. When a write occurs, it’s not considered successful until a majority of the nodes in the cluster have acknowledged it. This guarantees that all clients will see the same data, preventing inconsistencies that can plague distributed systems. This contrasts with eventually consistent systems, where updates might take some time to propagate.
• Data Model: etcd stores data in a hierarchical directory structure, similar to a file system. It uses a simple data model where each key can have one or more values, and changes to the data are versioned.

1.1.2. etcd’s Role in Distributed Systems (Service Discovery, Configuration Management, Leader Election):

etcd’s properties make it ideal for several critical tasks in distributed systems:

• Service Discovery: In a microservices architecture, services need a way to find each other. etcd acts as a central registry. Services register their location (IP address and port) in etcd when they start up. Other services can then query etcd to discover the locations of the services they need to communicate with. This dynamic discovery is essential in cloud environments where instances can come and go.
• Configuration Management: etcd stores configuration data that needs to be shared across multiple applications or instances. Instead of hardcoding configuration within each application, you can store it centrally in etcd. Applications can watch for changes in etcd and automatically update their configuration when it changes. This provides a single source of truth and simplifies updates. Examples include feature flags, database connection strings, and API keys.
• Leader Election: In many distributed systems, you need a single “leader” to coordinate tasks or manage resources. etcd provides mechanisms (using its consensus algorithm) to allow a group of processes to elect a leader. If the leader fails, a new leader is automatically elected. This ensures that there’s always a single point of coordination.
• Distributed Locks: etcd can implement distributed locks to allow different processes running to coordinate about the changes.

1.1.3. Why etcd is Crucial for Kubernetes:

Kubernetes relies on etcd as its primary datastore. It’s not an exaggeration to say that etcd is the “brain” of a Kubernetes cluster. Here’s why:

• Cluster State: Everything about the state of a Kubernetes cluster is stored in etcd: the definitions of Pods, Deployments, Services, ConfigMaps, Secrets, Nodes, and all other Kubernetes resources. When you run kubectl get pods, the information comes from etcd.
• Scheduling: The Kubernetes scheduler uses information from etcd to decide where to place new Pods. It considers resource availability, node affinity, and other constraints, all stored in etcd.
• API Server: The Kubernetes API server is the primary interface for interacting with the cluster. It acts as a front-end to etcd, providing authentication, authorization, and validation of requests before they are written to or read from etcd.
• Controller Manager: Kubernetes controllers (like the Deployment controller or ReplicaSet controller) constantly watch etcd for changes to resources. When a change occurs (e.g., you update a Deployment), the controller takes action to reconcile the actual state with the desired state, all based on information stored in etcd.
• Kubelet: kubelet uses etcd to get information about what to run.
• If etcd becomes unavailable or corrupted, the Kubernetes cluster will cease to function correctly. Existing Pods might continue to run, but you won’t be able to create new resources, scale existing ones, or make any changes to the cluster’s state. This highlights the critical importance of etcd’s reliability and availability.

1.2 Challenges of Managing etcd Clusters

While etcd is powerful, managing it directly can be complex, especially in production environments. Here are some of the key challenges:

1.2.1 Manual Setup and Configuration Complexity:

• Bootstrapping: Setting up a new etcd cluster from scratch involves several steps: generating TLS certificates (for secure communication), configuring each etcd member with the correct addresses and initial cluster settings, and starting the etcd processes. This process can be error-prone and time-consuming.
• Configuration Files: etcd is configured using command-line flags or a configuration file. Managing these configurations across multiple nodes and ensuring consistency can be difficult. Incorrect configurations can lead to cluster instability or data loss.
• Clustering: Configuring the cluster topology (number of members, their roles) requires a good understanding of etcd’s internals and the Raft consensus algorithm.

1.2.2 Backup and Restore Procedures:

• Data Loss Risk: etcd data is critical; losing it means losing your Kubernetes cluster state. Regular backups are essential, but implementing a robust backup and restore strategy can be challenging.
• Manual Snapshots: Taking snapshots of the etcd data directory is a common backup method. However, this needs to be done consistently across all members and requires careful handling to avoid data corruption.
• Restore Complexity: Restoring an etcd cluster from a backup involves multiple steps, including stopping the etcd processes, restoring the data directory, and restarting the cluster with the correct configuration. This process can be complex and requires careful planning to minimize downtime.
• Storage: Backups need to be stored securely and reliably. This often involves integrating with object storage services (like AWS S3, Azure Blob Storage, or Google Cloud Storage).

1.2.3 Scaling and Upgrading etcd:

• Scaling Out: Adding new members to an etcd cluster requires careful coordination to ensure that the new member joins the cluster correctly and doesn’t disrupt existing operations.
• Scaling In: Removing members also requires careful planning to avoid data loss or quorum issues.
• Rolling Upgrades: Upgrading etcd to a new version should be done without downtime. This typically involves a rolling upgrade, where each member is upgraded one at a time while the others remain online. This requires careful orchestration and monitoring.
• Version Compatibility: There could be breaking change between major version upgrade, so it should be planned accordingly.

1.2.4 Monitoring and Alerting:

• Cluster Health: Continuously monitoring the health and performance of the etcd cluster is crucial. This includes tracking metrics like latency, throughput, and the number of leader changes.
• Alerting: Setting up alerts to notify administrators of potential issues (e.g., high latency, disk space exhaustion, or leader election problems) is essential for proactive problem resolution.
• Integration: Integrating etcd monitoring with existing monitoring and alerting systems (like Prometheus and Grafana) can be complex.

1.3 Introduction to etcd-druid: The etcd Operator for Kubernetes

1.3.1 What is a Kubernetes Operator?

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application. More specifically, it’s a custom controller that extends the Kubernetes API to manage a specific application or service. Operators use Custom Resource Definitions (CRDs) to define the desired state of the application, and they continuously work to reconcile the actual state with the desired state. They automate tasks that would otherwise require manual intervention by a human operator. Key features of Operators:

• Automation: Operators automate complex operational tasks like deployments, upgrades, scaling, backups, and recovery.
• Custom Resources: They introduce new resource types (CRDs) that are specific to the application they manage.
• Reconciliation Loop: Operators continuously monitor the state of the application and take action to ensure it matches the desired state defined in the custom resources.
• Domain-Specific Knowledge: Operators encapsulate the operational knowledge required to manage a specific application.

1.3.2 How etcd-druid Simplifies etcd Management:

etcd-druid is an etcd Operator specifically designed to simplify the management of etcd clusters within Kubernetes. It addresses the challenges outlined in section 1.2 by automating many of the complex tasks involved in running etcd:

• Automated Deployment: etcd-druid automates the process of deploying etcd clusters. You define the desired state of your etcd cluster (number of members, storage, version, etc.) in an Etcd custom resource, and etcd-druid handles the rest. It creates the necessary Kubernetes resources (Pods, Services, PersistentVolumeClaims) and configures etcd correctly.
• Automated Backups and Restores: etcd-druid simplifies backup and restore operations. You define a backup schedule and storage location in an EtcdBackup custom resource, and etcd-druid automatically takes snapshots of the etcd data and uploads them to the specified storage. Restoring from a backup is also simplified through custom resources.
• Automated Scaling and Upgrades: Scaling an etcd cluster up or down is as simple as modifying the replicas field in the Etcd custom resource. etcd-druid handles the complexities of adding or removing etcd members. Similarly, upgrading to a new version of etcd can be done by updating the version field, and etcd-druid performs a rolling upgrade.
• Monitoring and Alerting (with external tools): While etcd-druid doesn't include built-in monitoring dashboards, it exposes metrics that can be easily integrated with monitoring tools like Prometheus and Grafana. This allows you to monitor the health and performance of your etcd clusters and set up alerts for potential issues.
• Automated Compaction: etcd-druid automates etcd data compaction via EtcdCopyBackupsTask resource to prevent performance degradation.

1.3.3 Key Benefits of Using etcd-druid:

• Reduced Operational Overhead: etcd-druid significantly reduces the time and effort required to manage etcd clusters.
• Improved Reliability: Automation reduces the risk of human error, leading to more reliable etcd deployments.
• Increased Efficiency: etcd-druid allows you to focus on your applications rather than the underlying infrastructure.
• Simplified Disaster Recovery: Automated backups and restores make it easier to recover from failures.
• Best Practices: etcd-druid embodies best practices for running etcd in Kubernetes.
• Declarative Management: Using CRDs the etcd cluster is managed declaratively.

Chapter 2: etcd-druid Architecture and Design

2.1 Core Components

2.1.1 etcd-druid Controller: The Brain of the Operator

The etcd-druid controller is the central component of the operator. It's a Kubernetes controller that runs as a Deployment within your Kubernetes cluster. Its primary responsibilities include:

• Watching Resources: The controller continuously watches for changes to specific Kubernetes resources, primarily the Etcd custom resource (and also EtcdBackup and EtcdCopyBackupsTask). This "watching" is done using the Kubernetes API.
• Reconciliation Loop: When a change is detected (a new Etcd resource is created, an existing one is modified, or one is deleted), the controller's reconciliation loop is triggered. This loop compares the desired state (defined in the Etcd resource) with the actual state of the etcd cluster.
• Taking Action: Based on the comparison, the controller takes actions to bring the actual state in line with the desired state. This might involve creating new Pods, updating existing Pods, creating Services, configuring PersistentVolumeClaims, or performing other Kubernetes operations.
• Status Updates: The controller updates the status subresource of the Etcd resource to reflect the current state of the etcd cluster. This provides feedback to the user about the progress of operations and any errors encountered.
• Event Handling: The controller emits Kubernetes events to provide a record of significant actions and state transitions.

2.1.2 Custom Resource Definitions (CRDs): Etcd, EtcdBackup, EtcdCopyBackupsTask

• etcd-druid extends the Kubernetes API by introducing three Custom Resource Definitions (CRDs). CRDs are essentially custom resource types that you can define to manage your applications. These CRDs are the primary way you interact with etcd-druid:
• Etcd: This is the main CRD. An Etcd resource represents a single etcd cluster. The spec section of the Etcd resource defines the desired state of the cluster, including:
• replicas: The number of etcd members.
• version: The desired etcd version.
• storage: Configuration for persistent storage (size, storage class).
• etcd: etcd specific configuration, like environment variables (env)
• backup: Embedded backup configuration (can also be done with a separate EtcdBackup resource).
• And other options for TLS, resource limits, etc.
• Here’s a simplified example YAML:

apiVersion: etcd.gardener.cloud/v1beta1
kind: Etcd
metadata:
  name: my-etcd-cluster
  namespace: default
spec:
  replicas: 3
  version: 3.5.9
  storage:
    storageClassName: standard
    capacity: 10Gi
  backup:
    store:
      secretRef:
        name: my-backup-secret
    schedule: "0 */6 * * *" # Every 6 hours
    maxBackups: 5

2.1.3 Sidecar Containers: Backup/Restore and Compaction

• etcd-druid uses sidecar containers within the etcd Pods to handle backup/restore and compaction operations. A sidecar container is a container that runs alongside the main application container (in this case, the etcd container) within the same Pod. They share the same network namespace and volumes.
• Backup/Restore Sidecar: This sidecar is responsible for:
• Taking snapshots of the etcd data directory.
• Uploading the snapshots to object storage (S3, Azure Blob Storage, GCS, etc.).
• Downloading snapshots from object storage during a restore operation.
• Managing the backup lifecycle (deleting old backups according to the retention policy).
• The sidecar communicates with the etcd container via the shared volume.
• Compaction Sidecar (Indirectly via EtcdCopyBackupsTask): The compaction is trigger via EtcdCopyBackupsTask resource. When this resource is applied, etcd-druid creates a job. This job internally creates pod with sidecar container. Sidecar container restores data from latest backup. By doing that, all historical data is removed.

2.2 Control Flow and Reconciliation Loop

2.2.1 How etcd-druid Watches for Changes to Etcd Resources:

etcd-druid uses the Kubernetes API's "watch" mechanism to monitor for changes to Etcd, EtcdBackup and EtcdCopyBackupsTask resources. This is a highly efficient mechanism where the API server notifies the controller only when a relevant change occurs (create, update, delete). The controller doesn't need to constantly poll the API server.

2.2.1 The Reconciliation Process: Creating, Updating, and Deleting etcd Clusters:

The reconciliation loop is the core of the operator’s logic. Here’s a simplified overview:

1. Trigger: The loop is triggered when:

• An Etcd resource is created.
• An Etcd resource is updated (e.g., the number of replicas is changed).
• An Etcd resource is deleted.
• An EtcdBackup resources is created, updated or deleted.
• An EtcdCopyBackupsTask resource is created.

1. Get Desired State: The controller reads the spec section of the Etcd resource (or EtcdBackup / EtcdCopyBackupsTask) to determine the desired state.
2. Get Actual State: The controller queries the Kubernetes API to determine the actual state of the etcd cluster. This includes checking for existing Pods, Services, PersistentVolumeClaims, and their status.
3. Compare: The controller compares the desired state with the actual state.
4. Take Action:

• Create: If the Etcd resource is new, the controller creates the necessary Kubernetes resources (Deployment or StatefulSet, Service, PersistentVolumeClaims) to deploy the etcd cluster.
• Update: If the Etcd resource has been modified, the controller updates the existing resources to match the new desired state (e.g., scaling the Deployment/StatefulSet, updating the etcd version).
• Delete: If the Etcd resource has been deleted, the controller cleans up the associated Kubernetes resources. It can optionally take a final backup before deleting the cluster.
• Backup: If EtcdBackup resource is created or updated, the controller updates the cron job which takes backup.
• Compaction: If EtcdCopyBackupsTask resource is created, then controller creates a job to restore the backup.

5. Update Status: The controller updates the status section of the Etcd resource (or other resources) to reflect the current state and any errors encountered.

6. Repeat: The loop runs continuously, ensuring that the actual state always converges to the desired state.

2.2.2 Handling Failures and Maintaining Desired State:

The reconciliation loop is designed to be resilient to failures.

• Retries: If an operation fails (e.g., a Pod creation fails due to insufficient resources), the controller will retry the operation. Kubernetes controllers typically use exponential backoff to avoid overwhelming the API server.
• Idempotency: Operations are designed to be idempotent, meaning that they can be applied multiple times without changing the outcome beyond the initial application. This ensures that the controller can safely retry operations without causing unintended side effects.
• Event Reporting: The controller emits Kubernetes events to record any errors or significant state transitions. This helps with debugging and monitoring.
• Status Updates: The status field of the Etcd resource provides information about the current state of the cluster, including any errors.

2.3 Data Storage and Persistence

2.3.1 How etcd-druid Manages etcd Data Volumes (Persistent Volumes):

etcd-druid uses Kubernetes Persistent Volumes (PVs) and PersistentVolumeClaims (PVCs) to manage etcd data storage. This ensures that etcd data persists even if the etcd Pods are restarted or rescheduled.

• PersistentVolumeClaim (PVC): When you create an Etcd resource, etcd-druid automatically creates a PVC for each etcd member. The PVC specifies the storage requirements (size, storage class) for the etcd data.
• PersistentVolume (PV): The Kubernetes cluster’s storage provisioner (e.g., a cloud provider’s storage service or a local storage provisioner) dynamically creates a PV to satisfy the PVC. The PV represents the actual storage volume.
• Mounting: The PV is mounted into the etcd Pod at the appropriate path (typically /var/etcd/data), allowing the etcd process to read and write its data.
• StatefulSet: etcd-druid typically uses a StatefulSet to manage the etcd Pods. StatefulSets are designed for stateful applications like databases and provide stable network identities and persistent storage for each Pod. Each Pod in a StatefulSet gets its own PVC, ensuring that data is preserved across Pod restarts.

2.3.2 Backup Storage Options (Object Storage: AWS S3, Azure Blob Storage, Google Cloud Storage, OpenStack Swift, Alicloud OSS):

• etcd-druid supports storing etcd backups in various object storage services. This provides a reliable and cost-effective way to store backups off-cluster, protecting against data loss in case of a complete cluster failure. Supported options include:
• AWS S3
• Azure Blob Storage
• Google Cloud Storage (GCS)
• OpenStack Swift
• Alicloud OSS
• Local (for testing, not recommended for production).
• And other S3 compatible storages.
• To configure backup storage, you need to:

1. Create a Kubernetes Secret: The Secret contains the credentials for accessing the object storage service (e.g., AWS access key ID and secret access key).
2. Specify the Storage Location: In the EtcdBackup resource (or in the backup section of the Etcd resource), you specify the object storage details:

• store.secretRef.name: The name of the Secret containing the credentials.
• store.prefix: The path (or "bucket" in S3 terminology) where the backups will be stored. This often includes a unique identifier for the etcd cluster to avoid conflicts.
• store.container: Specifies the container name for the object store.
• store.provider: Specifies the object store provider.

2.4 Networking

2.4.1 Service Exposure (Headless Services, ClusterIP):

Headless Service: etcd-druid creates a Headless Service for the etcd cluster. A Headless Service does not have a ClusterIP. Instead, DNS resolves the service name to the IP addresses of the individual Pods that are part of the service. This is crucial for etcd because clients need to be able to connect to individual etcd members (for example, to participate in leader election).

apiVersion: v1
kind: Service
metadata:
  name: my-etcd-cluster
  namespace: default
  labels:
    app.kubernetes.io/name: etcd
    app.kubernetes.io/instance: my-etcd-cluster
spec:
  ports:
  - name: etcd
    port: 2379
    targetPort: etcd
  - name: etcd-peer
    port: 2380
    targetPort: etcd-peer
  clusterIP: None  # This makes it a Headless Service
  selector:
    app.kubernetes.io/name: etcd
    app.kubernetes.io/instance: my-etcd-cluster

ClusterIP Service (Optional): etcd-druid can optionally create a ClusterIP Service. A ClusterIP Service provides a single, stable IP address within the Kubernetes cluster that can be used to access the etcd cluster. This is useful for clients that don't need to connect to individual members. However, direct access to individual members (via the Headless Service) is generally preferred for etcd clients.

2.4.2 Client Communication:

Clients (including the Kubernetes API server and other applications) connect to the etcd cluster using the service name and port.

• Kubernetes API Server: The Kubernetes API server is configured to connect to the etcd cluster using the Headless Service name and port (typically my-etcd-cluster.default.svc.cluster.local:2379). It uses TLS client certificates for authentication and encryption.
• Other Clients: Other applications can also connect to etcd. They typically use a client library (like etcdv3 for Go) and connect to the Headless Service name.
• Ports: The default port for client communication is 2379, and for peer communication is 2380

2.5 Security Considerations

2.5.1 TLS Encryption (etcd Communication, Client Communication):

etcd-druid supports TLS encryption for both etcd peer communication (between etcd members) and client communication (between clients and etcd members). This is essential for securing etcd data in transit.

• Automatic TLS: etcd-druid can automatically generate TLS certificates for the etcd cluster using cert-manager, if it's installed in the cluster.
• Manual TLS: You can also provide your own TLS certificates (CA certificate, server certificate, and client certificate) via Kubernetes Secrets.
• Configuration: The Etcd resource has options to enable TLS and specify the Secrets containing the certificates.

apiVersion: etcd.gardener.cloud/v1beta1
   kind: Etcd
   metadata:
     name: my-etcd-cluster
     namespace: default
   spec:
     # ... other settings ...
     tls:
       serverTLSSecretRef:
         name: etcd-server-tls
       clientTLSSecretRef:
         name: etcd-client-tls

2.5.2 Role-Based Access Control (RBAC) within Kubernetes:

etcd-druid itself requires appropriate RBAC permissions within the Kubernetes cluster to function correctly. It needs permissions to:

• Create, update, and delete Pods, Services, PersistentVolumeClaims, and other Kubernetes resources.
• Watch for changes to Etcd, EtcdBackup and EtcdCopyBackupsTask resources.
• Access Secrets (for object storage credentials and TLS certificates).
• These permissions are typically granted through ClusterRoles and ClusterRoleBindings (or Roles and RoleBindings for namespaced deployments). The etcd-druid Helm chart usually includes the necessary RBAC configurations.

2.5.3 Authentication of etcd

• etcd supports static and jwt authentication.

Chapter 3: Using etcd-druid: Deployment and Configuration

3.1 Prerequisites

Before you can deploy and use etcd-druid, you need the following:

Kubernetes Cluster: A functioning Kubernetes cluster is required. You can use any of the following:

• Minikube: A single-node Kubernetes cluster that runs locally on your machine. Suitable for testing and development.
• Kind (Kubernetes in Docker): Another tool for running local Kubernetes clusters using Docker containers. Good for testing and CI/CD pipelines.
• Cloud Provider Kubernetes Service: A managed Kubernetes service from a cloud provider like:
• Google Kubernetes Engine (GKE)
• Amazon Elastic Kubernetes Service (EKS)
• Azure Kubernetes Service (AKS)
• DigitalOcean Kubernetes
• Other managed Kubernetes offerings

The cluster should be at a supported Kubernetes version (check the etcd-druid documentation for the currently supported versions). Generally, recent versions of Kubernetes are recommended.

kubectl Configured: The kubectl command-line tool must be installed and configured to interact with your Kubernetes cluster. This means you should have a valid kubeconfig file that allows kubectl to authenticate to your cluster. You can usually test this by running kubectl get nodes.

(Optional) Object Storage Credentials: If you plan to use backups (which is highly recommended for production), you’ll need credentials for an object storage service. This could be:

• AWS: Access Key ID and Secret Access Key for an IAM user with permissions to access an S3 bucket.
• Azure: Connection string or other credentials for accessing Azure Blob Storage.
• GCP: Service account key file (JSON) with permissions to access Google Cloud Storage.
• OpenStack Swift: Credentials for accessing a Swift container.
• Alicloud: Access Key ID and Secret Access Key for an account.
• Other S3 compatible storage

You’ll store these credentials in a Kubernetes Secret, which etcd-druid will use to access the object storage.

3.2 Deploying etcd-druid

etcd-druid can be deployed in a couple of ways. The recommended approach is using Helm.

Using Helm Charts (Recommended):

Helm is a package manager for Kubernetes. It simplifies the deployment and management of applications. etcd-druid provides an official Helm chart. This is the preferred and easiest way to deploy.

1. Add the Helm Repository:

helm repo add etcd-druid https://gardener.github.io/etcd-druid
helm repo update

2. Install etcd-druid:

helm install etcd-druid etcd-druid/etcd-druid \
    --namespace etcd-druid --create-namespace \
    --set replicaCount=1 # Optional: Adjust the number of etcd-druid replicas

3. Verify Installation:

kubectl get pods -n etcd-druid

You should see the etcd-druid controller pod running.

3.3 Creating an Etcd Custom Resource

Once etcd-druid is deployed, you can create an Etcd custom resource to deploy an etcd cluster.

Essential YAML Configuration (Example YAML):

Here’s a basic example of an Etcd resource YAML file:

apiVersion: etcd.gardener.cloud/v1beta1
kind: Etcd
metadata:
  name: my-etcd-cluster
  namespace: default # Or any namespace where you want to deploy etcd
spec:
  replicas: 3
  version: 3.5.9  # Specify the desired etcd version
  storage:
    storageClassName: standard # Use your cluster's default storage class
    capacity: 10Gi
  backup:
    store:
      secretRef:
        name: my-backup-secret # Secret containing object storage credentials
        # key: <key-in-secret> # optional: if not default key names are used.
      prefix: my-etcd-backups/cluster1 # The path in your object storage
      provider: s3 # or aws, gcs, azure, oss, openstackswift
      #container: <bucket-name> # required for non-s3 providers.
    schedule: "0 */6 * * *" # Run backup every 6 hours (cron syntax)
    maxBackups: 5 # Keep the last 5 backups

etcd.env: (Optional) Allows you to set environment variables for the etcd container, which can be used to customize etcd configuration flags. See the etcd documentation for available flags. Example:

etcd:
    env:
    - name: ETCD_AUTO_COMPACTION_RETENTION
      value: "12h"

resources: (Optional) Define the resource requests and limits for etcd container.

resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi

Apply the YAML:

kubectl apply -f my-etcd-cluster.yaml

Verify the deployment

kubectl get etcd -n default
kubectl get pods -n default -l app.kubernetes.io/instance=my-etcd-cluster

3.4 Configuring Backups

As shown in the example in section 3.3, backups can be configured directly within the Etcd resource. However, you can also use a separate EtcdBackup resource for more flexibility.

Creating an EtcdBackup Resource:

apiVersion: etcd.gardener.cloud/v1beta1
kind: EtcdBackup
metadata:
  name: my-etcd-backup
  namespace: default # Must be in the same namespace as the Etcd resource
spec:
  etcdRef:
    name: my-etcd-cluster # Reference to the Etcd resource
    namespace: default
  store:
    secretRef:
      name: my-backup-secret
    prefix: my-etcd-backups/cluster1
    provider: gcs # Google Cloud Storage example
    container: my-backup-bucket # The GCS bucket name
  schedule: "0 */12 * * *" # Every 12 hours
  maxBackups: 7
  fullSnapshotSchedule: "0 0 */7 * *" # Full snapshot every 7 days.

Specifying Object Storage Details (Example YAML):

The store section is crucial. It defines where backups are stored. The examples above demonstrate basic configuration. You'll need to create the my-backup-secret Secret with the appropriate credentials for your chosen object storage provider. For example, for AWS S3:

kubectl create secret generic my-backup-secret -n default \
    --from-literal=AWS_ACCESS_KEY_ID='YOUR_ACCESS_KEY' \
    --from-literal=AWS_SECRET_ACCESS_KEY='YOUR_SECRET_KEY'

3.5 Restoring from Backup

etcd-druid supports restoring etcd clusters from backups.

Restoring to a new Etcd resource:

The recommended method for restore is to restore to a new Etcd resource. This is safer than attempting an in-place restore.

Create a new Etcd resource YAML: Create a new YAML file for the restored cluster (e.g., my-restored-etcd.yaml). Crucially, set the spec.backup.store to point to the location of your backups, and add an initialCluster field. Also set spec.replicas to the same number of replicas as your backed-up cluster.

apiVersion: etcd.gardener.cloud/v1beta1
kind: Etcd
metadata:
  name: my-restored-etcd  # A *new* name
  namespace: default
spec:
  replicas: 3 # Must match original cluster's replica count
  version: 3.5.9 # Specify same version.
  storage:
    storageClassName: standard
    capacity: 10Gi # adjust if you need
  backup:
    store:
      secretRef:
        name: my-backup-secret
      prefix: my-etcd-backups/cluster1 # Point to your backup location
      provider: s3

  # This section tells etcd-druid to restore from a backup
  restore:
      snapshot: <snapshot-name> # optional, use if you want to restore specific snapshot

Get Snapshot name (Optional): If you want to restore a specific snapshot and not the latest. Use etcdctl tool to get the list of available backups.

# Run etcdctl command to get backup details.
ETCDCTL_API=3 etcdctl --endpoints=my-backup-secret \
snapshot list my-etcd-backups/cluster1

Apply YAML

kubectl apply -f my-restored-etcd.yaml

In-place restore operation:

In-place restore is not recommended for production environments, as it is riskier. If something goes wrong during the restore, you could lose your existing etcd data. If you must do an in-place restore:

1. Scale down other applications: Scale down the applications which use etcd as datastore.
2. Edit the existing Etcd resource: Add the restore section as shown in the "restore to new resource", but to your existing Etcd resource. This will trigger an in-place restore.
3. Monitor Carefully: Watch the etcd-druid logs and the status of the Etcd resource very closely during the restore process.
4. Scale up the applications: Once restore is successful, scale up the application.

It’s strongly recommended to test restore procedures thoroughly in a non-production environment before attempting them in production.

3.6 Scaling etcd Clusters

Scaling Up and Down via the Etcd Resource:

Scaling an etcd cluster managed by etcd-druid is straightforward:

Edit the Etcd resource:

kubectl edit etcd my-etcd-cluster -n default

3.7 Upgrading etcd Clusters

Performing Rolling Upgrades using etcd-druid:

etcd-druid supports rolling upgrades of etcd clusters, allowing you to upgrade to a new version without downtime.

3.8 Monitoring etcd-druid and etcd Clusters

Using Prometheus and Grafana:

The recommended way to monitor etcd-druid and the etcd clusters it manages is to use Prometheus and Grafana.

• Prometheus: Prometheus is a time-series database and monitoring system. It scrapes metrics from configured targets at regular intervals.
• Grafana: Grafana is a data visualization and dashboarding tool. It can query data from Prometheus and display it in interactive dashboards.

1. Deploy Prometheus and Grafana: If you don’t already have them, you’ll need to deploy Prometheus and Grafana to your Kubernetes cluster. There are Helm charts available for both.
2. Configure Prometheus to Scrape etcd-druid and etcd:

etcd-druid exposes metrics in the Prometheus format at /metrics on port `8080`

Chapter 4: Advanced etcd-druid Features

4.1 Customizing etcd Configuration

etcd-druid provides several ways to customize the configuration of your etcd clusters beyond the basic settings (replicas, storage, version).

4.1.1 Using etcd.spec.etcd.env to set etcd flags:

The most direct way to customize etcd’s behavior is to set etcd flags using environment variables. The etcd.spec.etcd.env field in the Etcd resource allows you to do this. This field is an array of key-value pairs that will be passed as environment variables to the etcd container.

apiVersion: etcd.gardener.cloud/v1beta1
kind: Etcd
metadata:
  name: my-etcd-cluster
  namespace: default
spec:
  replicas: 3
  version: 3.5.9
  storage:
    storageClassName: standard
    capacity: 10Gi
  etcd:
    env:
    - name: ETCD_AUTO_COMPACTION_RETENTION
      value: "12h" # Set auto-compaction retention to 12 hours
    - name: ETCD_QUOTA_BACKEND_BYTES
      value: "8589934592"  # Set quota to 8 GiB (in bytes)
    - name: ETCD_MAX_REQUEST_BYTES
      value: "10485760"  # limit request size, example is 10 MiB

4.1.2 Commonly Used Flags

• ETCD_AUTO_COMPACTION_RETENTION: Controls how long etcd retains old revisions of keys. Setting this appropriately is crucial to prevent etcd's storage from growing indefinitely. You can specify the retention in hours (h), minutes (m), or seconds (s).
• ETCD_QUOTA_BACKEND_BYTES: Sets the maximum size of the etcd database (in bytes). When the quota is exceeded, etcd will raise an alarm and stop accepting writes. The default is 2GiB and the maximum is 8GiB.
• ETCD_MAX_REQUEST_BYTES: Limits the size of requests to the etcd server. The default is 1.5 MiB.
• ETCD_HEARTBEAT_INTERVAL: Defines the frequency of heartbeats.
• ETCD_ELECTION_TIMEOUT: Determines the election timeout duration.
• Refer to the official etcd documentation for a complete list of available flags and their meanings: https://etcd.io/docs/

4.2 Disaster Recovery Strategies

Disaster recovery (DR) is a critical aspect of managing any production system, and etcd is no exception. etcd-druid facilitates several DR strategies.

4.2.1 Multi-Region Deployments:

For the highest level of availability and resilience, you can deploy your Kubernetes cluster (and therefore your etcd cluster) across multiple regions. This protects against regional outages.

• Kubernetes Federation (Deprecated): Kubernetes Federation (v1 and v2) was a mechanism for managing multiple Kubernetes clusters as a single logical cluster. This has been deprecated.
• Multi-Cluster Management Tools: Modern approaches to multi-cluster management involve tools like:
• Karmada: Allows you to manage multiple Kubernetes clusters and distribute workloads across them.
• Open Cluster Management (OCM): Another multi-cluster management tool.
• Service Mesh: Istio, Linkerd etc to manage the traffics.
• etcd-druid in Multi-Cluster: With a multi-cluster setup, you would typically deploy etcd-druid and an etcd cluster in each Kubernetes cluster. The application workloads would then be distributed across the clusters. You would need a strategy for synchronizing or replicating data between the etcd clusters, or have separate etcd clusters that store different parts of the overall application state.
• Cross-Cluster Backup and Restore:
• This involves backing up an etcd cluster in one Kubernetes cluster and restoring it to a different Kubernetes cluster. This can be used to recover from a complete failure of the primary cluster.

1. Backup in Primary Cluster: Configure regular backups using etcd-druid in your primary Kubernetes cluster, storing the backups in a location accessible from both clusters (e.g., a shared S3 bucket).
2. Deploy etcd-druid in Secondary Cluster: Deploy etcd-druid to your secondary (recovery) Kubernetes cluster.
3. Restore to New Cluster: Create an Etcd resource in the secondary cluster, configuring the backup.store to point to the backup location in the shared storage. Use restore block to restore from backup. This will create a new etcd cluster in the secondary cluster, populated with the data from the backup.
4. Update DNS/Load Balancer: After the restore, you’ll need to update your DNS records or load balancer configuration to point to the new etcd cluster (and the services that depend on it) in the secondary cluster.

• This approach provides a way to recover your entire Kubernetes cluster state (or a specific application’s state) in a different cluster.

4.3 Troubleshooting Common Issues

4.3.1 Diagnosing etcd-druid Failures:

• Check etcd-druid Pod Logs: The first step in troubleshooting etcd-druid is to examine the logs of the etcd-druid controller pod:

kubectl logs -n etcd-druid <etcd-druid-pod-name>

• Check etcd-druid Events: Kubernetes events can provide valuable information about what etcd-druid is doing and any problems it has encountered:

kubectl get events -n etcd-druid

• Check Etcd Resource Status: The status field of the Etcd resource provides information about the current state of the etcd cluster:

kubectl get etcd my-etcd-cluster -n default -o yaml

Chapter 5: etcd-druid Competitors and Alternatives

5.1 Comparison with other etcd Operators

5.1.1 CoreOS (now Red Hat) etcd Operator:

The original etcd operator was developed by CoreOS (now part of Red Hat). It was one of the first operators and served as an inspiration for many others. However, it is now considered deprecated in favor of the Operator Lifecycle Manager (OLM) and the etcd operator available through OperatorHub.io.

5.1.2 Key Features (Historically):

• Automated etcd cluster provisioning and management.
• Backup and restore functionality.
• Rolling upgrades.

5.1.3 Differences from etcd-druid:

• Maintenance Status: The original CoreOS etcd operator is no longer actively maintained. etcd-druid is actively maintained by Gardener.
• Backup Approach: etcd-druid uses a sidecar container for backup and restore, while the CoreOS operator used a different approach.
• Focus: The CoreOS operator was more general-purpose, while etcd-druid is more tightly integrated with the Gardener ecosystem (though it can be used independently).
• Compaction Support: etcd-druid has dedicated EtcdCopyBackupsTask CRD for compaction, the CoreOS operator did not have this feature.
• Availability: Can be deployed via Operator Lifecycle Manager (OLM).

5.1.4 Other Community-Developed Operators:

• There are other community-developed etcd operators available, but they often lack the maturity, features, and active maintenance of etcd-druid. It's important to evaluate any community operator carefully, considering:
• Active Maintenance: Is the project actively maintained? Check the commit history and issue tracker.
• Community Support: Is there an active community around the operator? Are there users who can help with problems?
• Documentation: Is the operator well-documented?
• Features: Does the operator have the features you need (backup/restore, scaling, upgrades, etc.)?
• Security: Check the security aspects and reported vulnerabilities.

5.2 Managed etcd Services

Major cloud providers offer managed etcd services as part of their Kubernetes offerings (e.g., GKE, EKS, AKS). These services provide a fully managed etcd control plane for your Kubernetes cluster.

• Cloud Provider Offerings (e.g., AWS, Azure, GCP):
• Google Kubernetes Engine (GKE): GKE manages the etcd cluster for you. You don’t have direct access to the etcd instances.
• Amazon Elastic Kubernetes Service (EKS): Similar to GKE, EKS manages the etcd cluster as part of the control plane.
• Azure Kubernetes Service (AKS): AKS also manages the etcd cluster.

5.3 Manual etcd cluster management without operators

It’s possible to deploy and manage etcd clusters manually, without using an operator. This involves:

1. Provisioning Infrastructure: Provisioning virtual machines or bare-metal servers to run the etcd members.
2. Installing etcd: Downloading and installing the etcd binaries on each machine.
3. Generating TLS Certificates: Creating TLS certificates for secure communication between etcd members and clients.
4. Configuring etcd: Creating etcd configuration files with the correct settings (cluster topology, addresses, TLS certificates, etc.).
5. Starting etcd: Starting the etcd processes on each machine.
6. Setting up Monitoring and Alerting: Configuring monitoring and alerting for the etcd cluster.
7. Implementing Backup and Restore Procedures: Creating scripts or procedures to back up and restore the etcd data.
8. Handling Upgrades and Scaling: Manually performing rolling upgrades and scaling operations.

This approach is highly discouraged for production environments. It’s extremely complex, error-prone, and time-consuming. Operators like etcd-druid exist to automate these tasks and reduce the risk of human error. The manual approach is only suitable for learning purposes or very specific, highly customized scenarios where an operator doesn't provide the required flexibility.

This post is based on interaction with https://gemini.google.com/.

Happy learning :-)