Open in app

Sign in

Write

Sign in

Kubernetes operator in golang

Dilip Kumar
7 min readDec 9, 2024

--

Kubernetes Operator is essentially a software extension that automates the operation of a specific application on Kubernetes. It watches the state of your application, compares it to a desired state, and then takes actions to reconcile any differences.

Key Components of a Kubernetes Operator

  1. Controller: The core component that watches the state of your application. It uses Kubernetes’ API to query the current state and compare it to the desired state.
  2. Custom Resource Definition (CRD): Defines the desired state of your application in a Kubernetes-native way.
  3. Domain-Specific Logic: The logic that handles the reconciliation process, which involves creating, updating, or deleting Kubernetes resources to bring the actual state in line with the desired state.

Building a Kubernetes Operator in Golang

We’ll use the controller-runtime library to simplify the development process. Here's a basic structure:

package main

import (
        "context"

        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/log/zap"

        "myoperator/api/v1alpha1" // Replace with your CRD package
        "myoperator/controllers"
)

func main() {
        setupLog := zap.SetupWithOptions(zap.Development())
        ctrl.SetLogger(setupLog)
        // Manager
        mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
                Scheme:                 scheme,
                MetricsBindAddress:     "0.0.0.0:8080",
                Port:                   9443,
                LeaderElection:         true,
                LeaderElectionID:       "my-operator-leader-election",
        })
        if err != nil {
                setupLog.Error(err, "unable to start manager")
                os.Exit(1)
        }
        // Controller
        if err = (&controllers.MyController{
                Client: mgr.GetClient(),
                Scheme: mgr.GetScheme(),
        }).SetupWithManager(mgr); err != nil {
                setupLog.Error(err, "unable to setup controller")
                os.Exit(1)
        }

        if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
                setupLog.Error(err, "problem running manager")
                os.Exit(1)
        }
}

Breaking Down the Code:

  1. Manager:
  • Initializes the controller-runtime manager.
  • Sets up the scheme for registering CRDs.
  • Configures metrics and leader election.

2. Controller:

  • Defines the reconciliation logic for your CRD.
  • Watches for changes in the CRD’s status.
  • Reconciles the actual state with the desired state.

Writing the Controller Logic:

// controllers/my_controller.go

type MyController struct {
        client client.Client
        scheme *runtime.Scheme
}
func (r *MyController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        log := log.FromContext(ctx)

        // Fetch the MyResource instance
        myResource := &v1alpha1.MyResource{}
        err := r.Client.Get(ctx, req.NamespacedName, myResource)
        if err != nil {
                if errors.IsNotFound(err) {
                        // Resource not found, return. Could be deleted or has not been created.
                        return ctrl.Result{}, nil
                }
                return ctrl.Result{}, err
        }

        // Your reconciliation logic here
        // ...

        return ctrl.Result{}, nil
}

Custom Resource Definition (CRD)

This defines the schema for your custom resource, which users can then create to specify their desired state. Here’s a simplified example of a CRD YAML file:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresources.example.com
spec:
  group: example.com
  names:
    kind: MyResource
    plural: myresources
    singular: myresource
  scope: Namespaced
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer

Once you’ve defined your CRD, users can create instances of your custom resource like this:

apiVersion: example.com/v1alpha1
kind: MyResource
metadata:
  name: my-resource
spec:
  replicas: 3

When a user creates this custom resource, your operator’s controller will be triggered to reconcile the actual state (e.g., the number of pods) with the desired state (3 replicas).

Key Libraries for Kubernetes Operator Development in Golang

  • controller-runtime: This is the most widely used library for building Kubernetes Operators. It provides a framework for managing controllers, handling events, and reconciling resources.
  • client-go: This library provides a low-level client library for interacting with the Kubernetes API. While powerful, it can be more complex to use directly.
  • kubebuilder: This is a tool built on top of controller-runtime that provides a scaffolding tool and code generators to streamline the development process.
  • k8s.io/apimachinery: This library provides utilities for working with Kubernetes API objects, including custom resources.

Leveraging Existing YAML Files with Operator

While it’s possible to directly embed YAML manifests within your operator’s code, it’s generally recommended to separate the configuration from the code. This approach promotes better organization, reusability, and easier updates.

Here’s a suggested approach:

Leveraging Existing YAML Files with a Kubernetes Operator

Understanding the Approach:

While it’s possible to directly embed YAML manifests within your operator’s code, it’s generally recommended to separate the configuration from the code. This approach promotes better organization, reusability, and easier updates.

Here’s a suggested approach:

  1. Store YAML Files Externally:
  • ConfigMaps: Use ConfigMaps to store the YAML files as key-value pairs.
  • External Files: Reference external files from a Git repository or a shared storage volume.

2. Fetch and Parse YAML Files:

  • Use the Kubernetes API to fetch the ConfigMap or read the external file.
  • Parse the YAML content using a library like gopkg.in/yaml.v2 to obtain the desired Kubernetes objects.

3. Apply the Objects to the Cluster:

  • Use the client-go library to create, update, or delete the Kubernetes objects.
package controllers

import (
        "context"

        "k8s.io/apimachinery/pkg/api/errors"
        "k8s.io/apimachinery/pkg/runtime"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/log"

        mysqlexamplev1alpha1 "my-operator/api/v1alpha1"
        corev1 "k8s.io/api/core/v1"
        appsv1 "k8s.io/api/apps/v1"
        "gopkg.in/yaml.v2"
)

// MySQLReconciler reconciles a MySQL object
type MySQLReconciler struct {
        client.Client
        Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods;services;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete

func (r *MySQLReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        _ = log.FromContext(ctx)

        // ... (Fetch the MySQL instance)

        // Fetch the YAML manifests from ConfigMap or external file
        var deployment appsv1.Deployment
        var service corev1.Service
        // ... (Code to fetch and parse YAML)

        // Apply the Deployment and Service to the cluster
        err := r.Client.Create(ctx, &deployment)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }
        err = r.Client.Create(ctx, &service)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }

        // ... (Update the MySQL CRD Status)

        return ctrl.Result{}, nil
}

Using External YAML Files with a Kubernetes Operator

This approach involves storing your YAML files externally, such as in a Git repository or a shared storage volume. The operator will then fetch and parse these files during the reconciliation process.

package controllers

import (
        "context"
        "io/ioutil"
        "os"

        "k8s.io/apimachinery/pkg/api/errors"
        "k8s.io/apimachinery/pkg/runtime"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/log"

        mysqlexamplev1alpha1 "my-operator/api/v1alpha1"
        appsv1 "k8s.io/api/apps/v1"
        corev1 "k8s.io/api/core/v1"
        "gopkg.in/yaml.v2"
)

// MySQLReconciler reconciles a MySQL object
type MySQLReconciler struct {
        client.Client
        Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods;services;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete

func (r *MySQLReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        _ = log.FromContext(ctx)

        // ... (Fetch the MySQL instance)

        // Fetch the YAML manifests from external file
        deploymentYAML, err := ioutil.ReadFile("deployment.yaml")
        if err != nil {
                return ctrl.Result{}, err
        }
        serviceYAML, err := ioutil.ReadFile("service.yaml")
        if err != nil {
                return ctrl.Result{}, err
        }

        // Parse the YAML content
        var deployment appsv1.Deployment
        err = yaml.Unmarshal(deploymentYAML, &deployment)
        if err != nil {
                return ctrl.Result{}, err
        }
        var service corev1.Service
        err = yaml.Unmarshal(serviceYAML, &service)
        if err != nil {
                return ctrl.Result{}, err
        }

        // Apply the Deployment and Service to the cluster
        err = r.Client.Create(ctx, &deployment)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }
        err = r.Client.Create(ctx, &service)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }

        // ... (Update the MySQL CRD Status)

        return ctrl.Result{}, nil
}

Using ConfigMaps to Store YAML Files for Kubernetes Operators

This approach involves storing your YAML files as key-value pairs within a ConfigMap. The operator can then fetch and parse these files during the reconciliation process.

Here’s a basic outline of the steps involved:

  1. Create a ConfigMap:
  • Create a ConfigMap containing the YAML files as key-value pairs.

2. Fetch the ConfigMap:

  • Use the Kubernetes API to fetch the ConfigMap.

3. Parse the YAML Files:

  • Extract the YAML content from the ConfigMap.
  • Use a library like gopkg.in/yaml.v2 to parse the YAML content into Kubernetes object structures.

4. Apply Objects to the Cluster:

  • Use the client-go library to create, update, or delete the Kubernetes objects.
package controllers

import (
        "context"

        "k8s.io/apimachinery/pkg/api/errors"
        "k8s.io/apimachinery/pkg/runtime"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/log"

        mysqlexamplev1alpha1 "my-operator/api/v1alpha1"
        appsv1 "k8s.io/api/apps/v1"
        corev1 "k8s.io/api/core/v1"
        "gopkg.in/yaml.v2"
)

// MySQLReconciler reconciles a MySQL object
type MySQLReconciler struct {
        client.Client
        Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mysqlexample.com,resources=my-sqls/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods;services;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete

func (r *MySQLReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        _ = log.FromContext(ctx)

        // ... (Fetch the MySQL instance)

        // Fetch the ConfigMap
        configMap := &corev1.ConfigMap{}
        err := r.Get(ctx, client.ObjectKey{Namespace: req.Namespace, Name: "my-mysql-config"}, configMap)
        if err != nil {
                return ctrl.Result{}, err
        }

        // Parse the YAML content from the ConfigMap
        deploymentYAML := configMap.Data["deployment.yaml"]
        serviceYAML := configMap.Data["service.yaml"]

        var deployment appsv1.Deployment
        err = yaml.Unmarshal([]byte(deploymentYAML), &deployment)
        if err != nil {
                return ctrl.Result{}, err
        }
        var service corev1.Service
        err = yaml.Unmarshal([]byte(serviceYAML), &service)
        if err != nil {
                return ctrl.Result{}, err
        }

        // Apply the Deployment and Service to the cluster
        err = r.Client.Create(ctx, &deployment)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }
        err = r.Client.Create(ctx, &service)
        if err != nil && !errors.IsAlreadyExists(err) {
                return ctrl.Result{}, err
        }

        // ... (Update the MySQL CRD Status)

        return ctrl.Result{}, nil
}

Framework to build K8 operator

There are two primary frameworks for building Kubernetes Operators in Go:

1. Kubebuilder (https://book.kubebuilder.io/)

Key Features:

  • Simplifies the development process by generating boilerplate code.
  • Provides a strong foundation for building custom controllers and CRDs.
  • Offers tools for testing, linting, and building operators.
  • Integrates well with other Kubernetes tools and frameworks.

Best Suited For:

  • Developers who prefer a more hands-on approach and want fine-grained control over the operator’s behavior.
  • Advanced users who want to customize the operator’s functionality.

2. Operator SDK (https://github.com/operator-framework/operator-sdk)

Operator SDK is developed by a team of engineers from Red Hat and later open sourced.

Key Features:

  • Provides a higher-level abstraction for building operators.
  • Offers a comprehensive set of tools for managing the operator’s lifecycle.
  • Supports different operator types, including Ansible and Helm operators.
  • Integrates with OLM (Operator Lifecycle Manager) for easy deployment and management.

Best Suited For:

  • Developers who want a more streamlined and opinionated approach to operator development.
  • Those who are already familiar with Ansible or Helm.
  • Beginners who want to quickly get started with operator development.

Kubernetes controller-runtime Project

The Kubernetes controller-runtime Project is a set of go libraries for building Controllers. It is leveraged by Kubebuilder and Operator SDK.

Enjoy learning Kubernetes :-)

Kubernetes Operator

--

--

Dilip Kumar
Dilip Kumar

Written by Dilip Kumar

492 Followers
29 Following

With 18+ years of experience as a software engineer. Enjoy teaching, writing, leading team. Last 4+ years, working at Google as a backend Software Engineer.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams