Open in app

Sign in

Write

Sign in

Kubernetes Webhooks

Dilip Kumar
11 min readJan 13, 2025

--

A Kubernetes webhook is a mechanism that allows you to extend and customize the behavior of the Kubernetes API server.

Webhooks enable you to intercept and modify requests to the API server at various stages of processing, such as during admission control or authentication/authorization. This allows you to enforce custom policies, validate resources, mutate resources, or integrate with external systems.

Types of Kubernetes Webhooks

There are two main types of webhooks in Kubernetes:

  1. Admission Webhooks
  • These are used to intercept requests to the API server before they are persisted to the cluster.
  • There are two subtypes:
  • Validating Admission Webhooks: Used to validate incoming requests. For example, you can enforce custom rules to ensure that resources meet specific criteria before they are created or updated.
  • Mutating Admission Webhooks: Used to modify incoming requests. For example, you can automatically add default values or inject sidecar containers into Pod specifications.

2. Authentication/Authorization Webhooks

  • These are used to integrate external systems for authentication (e.g., verifying user credentials) or authorization (e.g., checking if a user has permission to perform an action).

How Webhooks Work?

  1. Configuration
  • You define a webhook configuration (e.g., ValidatingWebhookConfiguration or MutatingWebhookConfiguration) that specifies:
  • The API server endpoints to intercept (e.g., CREATE, UPDATE).
  • The URL of the external service (the webhook server) that will handle the request.
  • Rules to determine which resources and operations the webhook should apply to.

2. Webhook Server

  • You deploy a custom webhook server (e.g., a HTTP/HTTPS service) that receives requests from the Kubernetes API server.
  • The server processes the request, performs the necessary logic (e.g., validation or mutation), and returns a response to the API server.

3. API Server Interaction:

  • When a request matches the rules in the webhook configuration, the API server sends an admission review request to the webhook server.
  • The webhook server responds with an admission review response, which the API server uses to either allow, reject, or modify the request.

Example: Mutating Admission Webhook

Here’s an example of a MutatingWebhookConfiguration that injects a sidecar container into Pods:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar-injector.example.com
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  clientConfig:
    service:
      name: sidecar-injector
      namespace: default
      path: "/mutate"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Key Considerations

  • Webhook Server Availability: The webhook server must be highly available, as its failure can block API server operations.
  • Performance: Webhooks add latency to API requests, so ensure your webhook server is performant.
  • Security: Use HTTPS for webhook endpoints and validate requests to prevent spoofing or tampering.

Validating Admission Webhooks

Validating Admission Webhooks in Kubernetes allow you to enforce custom validation rules on resources before they are created or updated in the cluster. When a request is made to the API server (e.g., to create or update a resource), the API server sends the request to your custom webhook server for validation. The webhook server evaluates the request and either approves or rejects it based on your custom logic.

1. Webhook Server

The webhook server is an HTTP/HTTPS service that receives admission review requests from the Kubernetes API server and responds with whether the request should be allowed or denied.

Here’s an example of a simple webhook server written in Golang:

package main

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
)

// AdmissionReview represents the structure of an AdmissionReview request
type AdmissionReview struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Request    struct {
  UID    string `json:"uid"`
  Object struct {
   Metadata struct {
    Labels map[string]string `json:"labels"`
   } `json:"metadata"`
  } `json:"object"`
 } `json:"request"`
}

// AdmissionResponse represents the structure of an AdmissionReview response
type AdmissionResponse struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Response   struct {
  UID     string `json:"uid"`
  Allowed bool   `json:"allowed"`
  Status  struct {
   Message string `json:"message"`
  } `json:"status,omitempty"`
 } `json:"response"`
}

func validateHandler(w http.ResponseWriter, r *http.Request) {
 // Read the request body
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
  http.Error(w, "Failed to read request body", http.StatusBadRequest)
  return
 }
 defer r.Body.Close()

 // Parse the AdmissionReview request
 var admissionReview AdmissionReview
 if err := json.Unmarshal(body, &admissionReview); err != nil {
  http.Error(w, "Failed to parse AdmissionReview request", http.StatusBadRequest)
  return
 }

 // Extract the Pod labels
 labels := admissionReview.Request.Object.Metadata.Labels

 // Check if the required label exists
 if labels["env"] != "production" {
  response := AdmissionResponse{
   APIVersion: admissionReview.APIVersion,
   Kind:       admissionReview.Kind,
   Response: struct {
    UID     string `json:"uid"`
    Allowed bool   `json:"allowed"`
    Status  struct {
     Message string `json:"message"`
    } `json:"status,omitempty"`
   }{
    UID:     admissionReview.Request.UID,
    Allowed: false,
    Status: struct {
     Message string `json:"message"`
    }{
     Message: "All Pods must have the label 'env: production'",
    },
   },
  }

  // Send the response
  w.Header().Set("Content-Type", "application/json")
  if err := json.NewEncoder(w).Encode(response); err != nil {
   http.Error(w, "Failed to encode response", http.StatusInternalServerError)
  }
  return
 }

 // If the label is valid, allow the request
 response := AdmissionResponse{
  APIVersion: admissionReview.APIVersion,
  Kind:       admissionReview.Kind,
  Response: struct {
   UID     string `json:"uid"`
   Allowed bool   `json:"allowed"`
   Status  struct {
    Message string `json:"message"`
   } `json:"status,omitempty"`
  }{
   UID:     admissionReview.Request.UID,
   Allowed: true,
  },
 }

 // Send the response
 w.Header().Set("Content-Type", "application/json")
 if err := json.NewEncoder(w).Encode(response); err != nil {
  http.Error(w, "Failed to encode response", http.StatusInternalServerError)
 }
}

func main() {
 // Register the validateHandler function for the /validate endpoint
 http.HandleFunc("/validate", validateHandler)

 // Start the HTTPS server
 fmt.Println("Starting webhook server on :443...")
 if err := http.ListenAndServeTLS(":443", "server.crt", "server.key", nil); err != nil {
  log.Fatalf("Failed to start server: %v", err)
 }
}
  • The server listens for POST requests at the /validate endpoint.
  • It extracts the Pod specification from the admission review request and checks if the env: production label is present.
  • If the label is missing or incorrect, it returns a response denying the request with a message.
  • If the label is valid, it allows the request.

2. Deploy the Webhook Server

Deploy the webhook server to your Kubernetes cluster. You can use a Deployment and Service to expose the server.

Example Deployment and Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: validating-webhook-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: validating-webhook-server
  template:
    metadata:
      labels:
        app: validating-webhook-server
    spec:
      containers:
      - name: webhook-server
        image: my-webhook-server:latest
        ports:
        - containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: validating-webhook-server
spec:
  selector:
    app: validating-webhook-server
  ports:
  - port: 443
    targetPort: 443
  • my-webhook-server:latest is the image of webhook server.
  • The Service exposes the webhook server on port 443.

3. Create a ValidatingWebhookConfiguration

Define a ValidatingWebhookConfiguration to tell the Kubernetes API server to send requests to your webhook server for validation.

Example ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-label-validator
webhooks:
- name: pod-label-validator.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  clientConfig:
    service:
      name: validating-webhook-server
      namespace: default
      path: "/validate"
      port: 443
    caBundle: <base64-encoded-CA-certificate>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  • rules: Specifies which operations (e.g., CREATE, UPDATE) and resources (e.g., pods) the webhook should intercept.
  • clientConfig: Points to the webhook server’s Service and endpoint.
  • caBundle: Contains the base64-encoded CA certificate used to verify the webhook server’s TLS certificate.

4. Generate TLS Certificates

The webhook server must use HTTPS, so you need to generate TLS certificates for it. You can use tools like openssl or cert-manager to generate the certificates.

Example using openssl:

# Generate a private key
openssl genrsa -out server.key 2048

# Generate a certificate signing request (CSR)
openssl req -new -key server.key -out server.csr -subj "/CN=validating-webhook-server.default.svc"

# Generate a self-signed certificate
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# Encode the CA certificate in base64
cat server.crt | base64 | tr -d '\n'
  • Use the base64-encoded CA certificate in the caBundle field of the ValidatingWebhookConfiguration.

5. Test the Webhook

Create a Pod without the required label and verify that the webhook rejects it:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  labels:
    app: test
spec:
  containers:
  - name: nginx
    image: nginx

The webhook should deny the request with the message: All Pods must have the label 'env: production'.

Now, create a Pod with the required label:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  labels:
    env: production
spec:
  containers:
  - name: nginx
    image: nginx

The webhook should allow this request.

Mutating Admission Webhooks

Mutating Admission Webhooks in Kubernetes allow you to modify incoming requests to the API server before they are persisted to the cluster. For example, you can automatically inject sidecar containers, add annotations, or set default values for resources.

1. Create a Webhook Server in Go

The webhook server will intercept Pod creation requests, check if a specific sidecar container is present, and inject it if necessary.

Here’s the Go implementation:

package main

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
)

// AdmissionReview represents the structure of an AdmissionReview request
type AdmissionReview struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Request    struct {
  UID    string `json:"uid"`
  Object struct {
   Metadata struct {
    Name      string            `json:"name"`
    Namespace string            `json:"namespace"`
    Labels    map[string]string `json:"labels"`
   } `json:"metadata"`
   Spec struct {
    Containers []struct {
     Name  string `json:"name"`
     Image string `json:"image"`
    } `json:"containers"`
   } `json:"spec"`
  } `json:"object"`
 } `json:"request"`
}

// AdmissionResponse represents the structure of an AdmissionReview response
type AdmissionResponse struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Response   struct {
  UID       string `json:"uid"`
  Allowed   bool   `json:"allowed"`
  PatchType string `json:"patchType,omitempty"`
  Patch     string `json:"patch,omitempty"`
 } `json:"response"`
}

// Sidecar container definition
var sidecarContainer = map[string]interface{}{
 "name":  "sidecar",
 "image": "busybox:latest",
}

func mutateHandler(w http.ResponseWriter, r *http.Request) {
 // Read the request body
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
  http.Error(w, "Failed to read request body", http.StatusBadRequest)
  return
 }
 defer r.Body.Close()

 // Parse the AdmissionReview request
 var admissionReview AdmissionReview
 if err := json.Unmarshal(body, &admissionReview); err != nil {
  http.Error(w, "Failed to parse AdmissionReview request", http.StatusBadRequest)
  return
 }

 // Check if the sidecar container already exists
 hasSidecar := false
 for _, container := range admissionReview.Request.Object.Spec.Containers {
  if container.Name == "sidecar" {
   hasSidecar = true
   break
  }
 }

 // If the sidecar is missing, create a patch to add it
 var patch []map[string]interface{}
 if !hasSidecar {
  patch = append(patch, map[string]interface{}{
   "op":    "add",
   "path":  "/spec/containers/-",
   "value": sidecarContainer,
  })
 }

 // Marshal the patch into JSON
 patchBytes, err := json.Marshal(patch)
 if err != nil {
  http.Error(w, "Failed to marshal patch", http.StatusInternalServerError)
  return
 }

 // Prepare the AdmissionResponse
 response := AdmissionResponse{
  APIVersion: admissionReview.APIVersion,
  Kind:       admissionReview.Kind,
  Response: struct {
   UID       string `json:"uid"`
   Allowed   bool   `json:"allowed"`
   PatchType string `json:"patchType,omitempty"`
   Patch     string `json:"patch,omitempty"`
  }{
   UID:       admissionReview.Request.UID,
   Allowed:   true,
   PatchType: "JSONPatch",
   Patch:     string(patchBytes),
  },
 }

 // Send the response
 w.Header().Set("Content-Type", "application/json")
 if err := json.NewEncoder(w).Encode(response); err != nil {
  http.Error(w, "Failed to encode response", http.StatusInternalServerError)
 }
}

func main() {
 // Register the mutateHandler function for the /mutate endpoint
 http.HandleFunc("/mutate", mutateHandler)

 // Start the HTTPS server
 fmt.Println("Starting mutating webhook server on :443...")
 if err := http.ListenAndServeTLS(":443", "server.crt", "server.key", nil); err != nil {
  log.Fatalf("Failed to start server: %v", err)
 }
}

Rest of the steps is same as validating webhook exacpt that here we need Create the MutatingWebhookConfiguration as below sample.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar-injector.example.com
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  clientConfig:
    service:
      name: mutating-webhook-server
      namespace: default
      path: "/mutate"
      port: 443
    caBundle: <base64-encoded-CA-certificate>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Authentication/Authorization Webhooks

Authentication and Authorization Webhooks in Kubernetes allow you to integrate external systems for authentication (verifying user credentials) and authorization (checking if a user has permission to perform an action). These webhooks enable you to extend Kubernetes’ built-in authentication and authorization mechanisms with custom logic or external identity providers.

Authentication Webhook

  • Used to verify the identity of a user or service account.
  • The Kubernetes API server sends a TokenReview request to the external webhook server, which validates the provided token and returns the user information.

Authorization Webhook

  • Used to determine if a user or service account is allowed to perform a specific action on a resource.
  • The Kubernetes API server sends a SubjectAccessReview request to the external webhook server, which evaluates the request and returns whether it is allowed or denied.

Authentication Webhook Example in Go

Below is an example of an Authentication Webhook server in Go that validates a token and returns user information.

package main

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
)

// TokenReview represents the structure of a TokenReview request
type TokenReview struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Spec       struct {
  Token string `json:"token"`
 } `json:"spec"`
}

// TokenReviewResponse represents the structure of a TokenReview response
type TokenReviewResponse struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Status     struct {
  Authenticated bool              `json:"authenticated"`
  User          map[string]string `json:"user,omitempty"`
 } `json:"status"`
}

func authenticateHandler(w http.ResponseWriter, r *http.Request) {
 // Read the request body
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
  http.Error(w, "Failed to read request body", http.StatusBadRequest)
  return
 }
 defer r.Body.Close()

 // Parse the TokenReview request
 var tokenReview TokenReview
 if err := json.Unmarshal(body, &tokenReview); err != nil {
  http.Error(w, "Failed to parse TokenReview request", http.StatusBadRequest)
  return
 }

 // Validate the token (example: check if the token is "valid-token")
 token := tokenReview.Spec.Token
 authenticated := false
 user := make(map[string]string)

 if token == "valid-token" {
  authenticated = true
  user = map[string]string{
   "username": "admin",
   "uid":      "12345",
   "groups":   "system:masters",
  }
 }

 // Prepare the TokenReviewResponse
 response := TokenReviewResponse{
  APIVersion: tokenReview.APIVersion,
  Kind:       tokenReview.Kind,
  Status: struct {
   Authenticated bool              `json:"authenticated"`
   User          map[string]string `json:"user,omitempty"`
  }{
   Authenticated: authenticated,
   User:          user,
  },
 }

 // Send the response
 w.Header().Set("Content-Type", "application/json")
 if err := json.NewEncoder(w).Encode(response); err != nil {
  http.Error(w, "Failed to encode response", http.StatusInternalServerError)
 }
}

func main() {
 // Register the authenticateHandler function for the /authenticate endpoint
 http.HandleFunc("/authenticate", authenticateHandler)

 // Start the HTTPS server
 fmt.Println("Starting authentication webhook server on :443...")
 if err := http.ListenAndServeTLS(":443", "server.crt", "server.key", nil); err != nil {
  log.Fatalf("Failed to start server: %v", err)
 }
}

Authorization Webhook Example in Go

Below is an example of an Authorization Webhook server in Go that checks if a user is allowed to perform a specific action.

package main

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
)

// SubjectAccessReview represents the structure of a SubjectAccessReview request
type SubjectAccessReview struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Spec       struct {
  ResourceAttributes struct {
   Namespace string `json:"namespace"`
   Verb      string `json:"verb"`
   Resource  string `json:"resource"`
  } `json:"resourceAttributes"`
  User   string `json:"user"`
  Groups string `json:"groups"`
 } `json:"spec"`
}

// SubjectAccessReviewResponse represents the structure of a SubjectAccessReview response
type SubjectAccessReviewResponse struct {
 APIVersion string `json:"apiVersion"`
 Kind       string `json:"kind"`
 Status     struct {
  Allowed bool   `json:"allowed"`
  Reason  string `json:"reason,omitempty"`
 } `json:"status"`
}

func authorizeHandler(w http.ResponseWriter, r *http.Request) {
 // Read the request body
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
  http.Error(w, "Failed to read request body", http.StatusBadRequest)
  return
 }
 defer r.Body.Close()

 // Parse the SubjectAccessReview request
 var subjectAccessReview SubjectAccessReview
 if err := json.Unmarshal(body, &subjectAccessReview); err != nil {
  http.Error(w, "Failed to parse SubjectAccessReview request", http.StatusBadRequest)
  return
 }

 // Check if the user is allowed to perform the action
 allowed := false
 user := subjectAccessReview.Spec.User
 verb := subjectAccessReview.Spec.ResourceAttributes.Verb
 resource := subjectAccessReview.Spec.ResourceAttributes.Resource

 if user == "admin" && verb == "get" && resource == "pods" {
  allowed = true
 }

 // Prepare the SubjectAccessReviewResponse
 response := SubjectAccessReviewResponse{
  APIVersion: subjectAccessReview.APIVersion,
  Kind:       subjectAccessReview.Kind,
  Status: struct {
   Allowed bool   `json:"allowed"`
   Reason  string `json:"reason,omitempty"`
  }{
   Allowed: allowed,
   Reason:  "User is allowed to get pods",
  },
 }

 // Send the response
 w.Header().Set("Content-Type", "application/json")
 if err := json.NewEncoder(w).Encode(response); err != nil {
  http.Error(w, "Failed to encode response", http.StatusInternalServerError)
 }
}

func main() {
 // Register the authorizeHandler function for the /authorize endpoint
 http.HandleFunc("/authorize", authorizeHandler)

 // Start the HTTPS server
 fmt.Println("Starting authorization webhook server on :443...")
 if err := http.ListenAndServeTLS(":443", "server.crt", "server.key", nil); err != nil {
  log.Fatalf("Failed to start server: %v", err)
 }
}

Configure Kubernetes to Use the Webhooks

Update the Kubernetes API server configuration to point to the webhook servers.

Example configuration for authentication:

apiVersion: v1
kind: Config
clusters:
- name: webhook-server
  cluster:
    server: https://webhook-server.default.svc:443/authenticate
    certificate-authority: /path/to/ca.crt
users:
- name: webhook
  user:
    client-certificate: /path/to/client.crt
    client-key: /path/to/client.key
contexts:
- context:
    cluster: webhook-server
    user: webhook
  name: webhook
current-context: webhook

Example configuration for authorization:

apiVersion: v1
kind: Config
clusters:
- name: webhook-server
  cluster:
    server: https://webhook-server.default.svc:443/authorize
    certificate-authority: /path/to/ca.crt
users:
- name: webhook
  user:
    client-certificate: /path/to/client.crt
    client-key: /path/to/client.key
contexts:
- context:
    cluster: webhook-server
    user: webhook
  name: webhook
current-context: webhook

This post is based on interaction with https://chat.deepseek.com. Enjoy using LLM to enhance your learning :-)

--

--

Dilip Kumar
Dilip Kumar

Written by Dilip Kumar

492 Followers
29 Following

With 18+ years of experience as a software engineer. Enjoy teaching, writing, leading team. Last 4+ years, working at Google as a backend Software Engineer.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams