Prow a Kubernetes-native CI/CD

13 min readDec 10, 2024

What is Prow?

Prow is a Kubernetes-native CI/CD system that excels at automating workflows and managing jobs at scale. It’s the engine behind testing and deployment for many prominent projects like Kubernetes itself, Istio, and Kubeflow. Prow’s strength comes from its tight integration with Kubernetes, leveraging its scalability and reliability.

Prow was originally developed by the Kubernetes community, specifically within the Kubernetes SIG Testing group. This special interest group focuses on testing and improving the reliability of Kubernetes itself.

Since Prow proved to be such a powerful tool, it gradually evolved into a more general-purpose CI/CD system that can be used for projects beyond Kubernetes.

Key Features of Prow

Kubernetes-Native: Prow runs entirely within your Kubernetes cluster, using its resources to execute jobs. This means it scales seamlessly and benefits from Kubernetes’ robust architecture.
GitHub Integration: Prow interacts directly with GitHub, responding to events like pull requests, issues, and comments. This automation streamlines code reviews and deployments.
Job Execution: Prow defines jobs as Kubernetes pods, providing flexibility and control over your build and deployment processes.
ChatOps: Interact with Prow through chat platforms like Slack, triggering jobs or getting status updates with simple commands.
Plugins: Extend Prow’s functionality with plugins for linting, labeling, and other custom actions.

How to use Prow?

GitHub Webhooks: Prow listens for events on your GitHub repositories through webhooks. When you open a pull request or push code, GitHub notifies Prow.
Prow Plugins: Prow uses plugins to process these events. Plugins can trigger jobs, enforce policies, or automate tasks based on the event.
Job Execution: Prow creates Kubernetes pods to run your CI/CD jobs. These pods have all the necessary dependencies and configurations to build, test, and deploy your code.
Reporting: Prow reports job statuses back to GitHub, updating pull requests with results and logs.

Prow architecture

Prow is made up of a collection of microservices (aka “Prow components”) that work together in a service cluster, leveraging one or more build clusters to schedule Prow Jobs (or just “jobs”).

Prow creates jobs based on various types of events, such as:

GitHub events (e.g., a new PR is created, or is merged, or a person comments “/retest” on a PR),
Pub/Sub messages,
time (these are created by Horologium and are called periodic jobs), and
retesting (triggered by Tide).

Jobs are created inside the Service Cluster as Kubernetes Custom Resources. The Prow Controller Manager takes triggered jobs and schedules them into a build cluster, where they run as Kubernetes pods. Crier then reports the results back to GitHub.

Here’s a breakdown of the key components:

1. Prow Components (Microservices)

These are the core services that make Prow work:

Hook: This is the entry point for events from GitHub. It listens for webhooks and routes them to the appropriate plugins.
Plugins: These are small programs that extend Prow’s functionality. They can trigger jobs, enforce policies (like requiring certain labels on pull requests), or automate tasks based on events.
Crier: This component is responsible for reporting job statuses back to GitHub, adding comments to pull requests, and updating commit statuses.
Deck: This provides a user interface for Prow, allowing you to view job history, configurations, and other information.
Horologium: This handles scheduling periodic jobs, such as nightly builds or test runs.
Sinker: Cleans up old Prow jobs and pods to keep your cluster tidy.
Tide: This component merges approved pull requests automatically based on configurable merge policies.

2. Kubernetes Clusters

Prow typically operates across two types of Kubernetes clusters:

Service Cluster: This is where the Prow components themselves run. It’s the control center for your CI/CD system.
Build Cluster: This is where your actual CI/CD jobs are executed. You can have one or multiple build clusters, allowing you to isolate workloads or use different environments for testing.

3. Job Execution

Prow defines jobs as Kubernetes pods. When a job is triggered, Prow creates a pod in the build cluster with all the necessary dependencies and configurations to execute your tasks (building, testing, deploying, etc.).

How it all works together:

Event Trigger: An event occurs on GitHub (e.g., a new pull request).
Webhook to Hook: GitHub sends a webhook notification to the Hook component in the service cluster.
Plugin Processing: Hook passes the event to the relevant plugins.
Job Trigger: If a plugin decides to trigger a job, it sends a request to the Prow Job controller.
Pod Creation: The Prow Job controller creates a pod in the build cluster to execute the job.
Reporting: Crier reports the job status back to GitHub, updating the pull request with results and logs.

Write Prow Plugins

Prow plugins lets us to customize and extend Prow’s capabilities to match our specific CI/CD needs. Here’s a breakdown of how to create our own Prow plugins:

1. Choose Your Plugin Type

Prow plugins are categorized by the types of GitHub events they handle. Common types include:

IssueCommentEvent: Respond to comments on issues.
PullRequestEvent: React to pull request actions (opening, closing, merging, etc.).
PushEvent: Trigger actions when code is pushed to a repository.
StatusEvent: Respond to changes in commit statuses.

2. Set Up Your Development Environment

Go: Prow plugins are written in Go, so make sure you have Go installed and configured on your system.
Prow Dependencies: You’ll need to get the Prow source code to access the necessary libraries and interfaces for plugin development. You can clone the Prow repository from GitHub: git clone https://github.com/kubernetes/test-infra.git

3. Structure Your Plugin

Create a Package: Create a new Go package for your plugin within the prow/plugins directory in the Prow source code.
Implement the Handler: Define a function that implements the github.EventHandler interface for the event type you want to handle. This function will be called by Prow when the corresponding event occurs.
Register the Plugin: In your plugin’s init() function, use plugins.Register*Handler() (e.g., plugins.RegisterIssueCommentHandler) to register your handler with Prow.

4. Write the Plugin Logic

Access Event Data: Your handler function receives a github.Event object containing information about the event that triggered it.
Use the Plugin Client: Prow provides a plugins.PluginClient that gives you access to a variety of helpful clients and agents:
GitHub Client: Interact with the GitHub API (create comments, labels, etc.).
ProwJob Client: Trigger Prow jobs.
Git Client: Perform Git operations.
OWNERS Client: Access OWNERS files for repository permissions.
Slack Client: Send messages to Slack.
Implement Your Logic: Use the provided clients and event data to implement the desired behavior for your plugin.

5. Configure and Deploy

Add to plugins.yaml: Add your plugin to the plugins.yaml configuration file in your Prow instance to enable it for specific repositories.
Deploy: Build and deploy your updated Prow instance to apply the changes.

Simple Issue Comment Plugin

package myplugin

import (
    "fmt"
    "github.com/sirupsen/logrus"
    "k8s.io/test-infra/prow/github"
    "k8s.io/test-infra/prow/pluginhelp"
    "k8s.io/test-infra/prow/plugins"
)

func handleIssueComment(pc plugins.PluginClient, ice *github.IssueCommentEvent) error {
    if ice.Comment.Body == "/hello" {
        return pc.GitHubClient.CreateComment(
            ice.Repo.Owner.Login, ice.Repo.Name, ice.Issue.Number,
            fmt.Sprintf("Hello @%s!", ice.Comment.User.Login))
    }
    return nil
}

func init() {
    plugins.RegisterIssueCommentHandler("my-plugin", handleIssueComment, helpProvider)
}

func helpProvider(config *plugins.Configuration, enabledRepos []string) (*pluginhelp.PluginHelp, error) {
    // Provide help information for your plugin
    return &pluginhelp.PluginHelp{
            Description: "My plugin responds to '/hello' commands on issues.",
        }, nil
}

Install Prow

Here’s a general guide to installing Prow, keeping in mind that there might be slight variations depending on your specific environment and needs:

Prerequisites

Kubernetes Cluster: You’ll need a running Kubernetes cluster (version 1.19 or later is recommended). This could be a local cluster like Minikube or Kind, or a cloud-based cluster on GKE, AKS, or EKS.
kubectl: Make sure you have the kubectl command-line tool installed and configured to interact with your cluster.
GitHub Account: You’ll need a GitHub account and a repository where you want to use Prow.
Go: Prow is written in Go, so ensure you have Go installed on your system.

Installation Steps

Create a GitHub App

In your GitHub account, go to Settings -> Developer settings -> GitHub Apps -> New GitHub App.
Fill in the required details for your app (name, homepage URL, webhook URL, etc.).
Important: Under “Permissions & events,” grant the app the necessary permissions for the repositories you want to use with Prow (e.g., read access to code, pull requests, issues, and write access to commit statuses).
Generate a private key for your app and download it. You’ll need this later.

2. Deploy Prow to Your Cluster

Get the Prow manifests: You can use the prow/cluster/ directory in the Prow repository as a starting point for your Prow deployment.
Configure the manifests: Update the manifests with your specific settings, such as the GitHub app ID, private key, webhook secret, and the location where Prow should store its configuration and data (e.g., a GCS bucket or an Azure blob storage).
Apply the manifests: Use kubectl apply -f to deploy the Prow components (Hook, Deck, Crier, etc.) to your cluster.

3. Configure the Webhook

In your GitHub app settings, go to the “Webhook” section.
Set the “Payload URL” to the external IP address or hostname of your Hook service, along with the correct path (usually /hook).
Choose “Content type” as application/json.
Add the webhook secret you configured in the Prow manifests.
Select the events you want to trigger the webhook (e.g., pull requests, pushes, issue comments).

4. Install the App on Your Repository

In your GitHub app settings, go to the “Install App” section.
Choose the organization or repository where you want to install the app.
This will grant the app access to your repository and allow Prow to start processing events.

5. Configure Prow Components

config.yaml: This file defines the overall Prow configuration, including the location of your plugins and jobs configurations.
plugins.yaml: This file configures which plugins are enabled for which repositories.
Job config files: These files define your CI/CD jobs, specifying the container images, commands, and resources required for each job.

6. Access the Prow Deck UI

Find the service for the Prow Deck component in your cluster (e.g., kubectl get svc -n prow).
Access the Deck UI through the service’s external IP address or NodePort.
You can now use the Deck to view job history, configurations, and other Prow information.

Prow UI

Prow Deck component provides UI component.

Dashboard

the Prow Deck UI dashboard is your window into the activity and status of your Prow CI/CD system.

Here’s a breakdown of the key data it displays:

1. Job Status and History

List of Jobs: View a list of recent Prow jobs, including their names, repositories, and statuses (success, failure, pending, triggered).
Job Details: Drill down into individual jobs to see detailed information like:
a. Logs: View the logs from the job’s pods to diagnose errors or understand the execution flow.
b. Artifacts: Access any artifacts produced by the job (e.g., test results, build outputs).
c. GitHub Information: See the associated pull request or commit on GitHub.
d. Pod Details: Examine the Kubernetes pods that executed the job, including their resource usage and events.
Filtering and Sorting: Filter jobs by repository, status, author, and other criteria. Sort jobs by start time, completion time, etc.

2. ProwJob Configuration

Job Definitions: View the YAML configurations for your Prow jobs, showing how they are defined and the parameters they use.
Presubmits and Postsubmits: See how your jobs are configured to run as presubmits (before merging pull requests) or postsubmits (after merging).

3. Repository Activity

Pull Request Status: Get an overview of the status of pull requests in your repositories, including which jobs have passed or failed.
Branch Protection: See the branch protection rules configured for your repositories, which might include requiring certain Prow jobs to pass before merging.

4. Tide Information

Tide Pools: If you’re using Tide for automated merging, you can see the status of Tide pools, which are groups of pull requests that Tide is considering for merging.
Merge Conflicts: Identify any merge conflicts that are preventing pull requests from being merged.

5. Prow Component Status

Health: Get an indication of the health of the Prow components themselves (Hook, Crier, etc.).

Navigation and Features

Search: Search for specific jobs or repositories.
Links to GitHub: Easily navigate to the corresponding pull requests or issues on GitHub.
Log Streaming: Stream logs from running jobs in real time.
Configuration Visualization: Some Prow Deck instances might include visualizations of your Prow configuration to help you understand the relationships between jobs and repositories.

The Prow Deck UI provides a centralized location to monitor your CI/CD pipelines, troubleshoot issues, and gain insights into your Prow setup.

Rerun Prow Job via Prow UI

Rerun prow job can be done by visiting prow UI, locate prow job and rerun job by clicking on the ↻ button, selecting a configuration option, and then clicking Rerun button. For prow on github, the permission is controlled by github membership, and configured as part of deck configuration, see rerun_auth_configs for k8s prow.

See example below:

Rerunning can also be done on Spyglass:

This is also available for non github prow if the frontend is secured and allow_anyone is set to true for the job.

Abort Prow Job via Prow UI

Aborting a prow job can be done by visiting the prow UI, locate the prow job and abort the job by clicking on the ✕ button, and then clicking Confirm button. For prow on github, the permission is controlled by github membership, and configured as part of deck configuration, see rerun_auth_configs for k8s prow. Note, the abort functionality uses the same field as rerun for permissions.

See example below:

Aborting can also be done on Spyglass:

This is also available for non github prow if the frontend is secured and allow_anyone is set to true for the job.

Spyglass component

Spyglass is a key component in Prow that enhances how you view and interact with the artifacts produced by your CI/CD jobs. Think of it as Prow’s visualization engine.

Here’s what Spyglass does:

Artifact Viewing: Spyglass provides a way to view various types of artifacts generated by your Prow jobs. These artifacts could be anything from simple log files and test results to images, graphs, and HTML reports.
Customizable Rendering: Spyglass uses “lenses” to render artifacts in different ways. Lenses are essentially viewers or renderers tailored to specific artifact types. For example, a lens might display logs with syntax highlighting, render images in a gallery, or present test results in a structured format.
Centralized Access: Spyglass makes it easy to access artifacts from different jobs in a central location. This is particularly helpful when you have complex workflows with multiple jobs producing various outputs.

How Spyglass Works

Artifact Storage: When Prow jobs run, they typically store their artifacts in a persistent location, such as a cloud storage bucket (GCS, S3, etc.).
Spyglass Configuration: You configure Spyglass to know where to find these artifacts and which lenses to use for rendering them.
Artifact Discovery: When you want to view artifacts, Spyglass uses its configuration to locate the relevant files.
Lens Rendering: Spyglass matches the artifacts with appropriate lenses based on file types or patterns.
Visualization: The chosen lenses render the artifacts in a user-friendly way within the Prow Deck UI.

Benefits of Spyglass

Improved Visualization: Makes it easier to understand and analyze the results of your CI/CD jobs.
Enhanced Debugging: Helps you quickly identify issues and debug failures by providing clear visualizations of logs, test results, and other artifacts.
Customization: Allows you to create custom lenses to render artifacts in ways that are specific to your needs.
Centralized View: Provides a single place to access and view artifacts from all your Prow jobs.

Integration with Prow Deck

Spyglass is typically integrated with the Prow Deck UI. When you view a Prow job in Deck, you’ll often see a “Artifacts” tab or section that uses Spyglass to display the job’s outputs.

Use Prow for Continuous Deployment

Prow is quite versatile when it comes to continuous deployment (CD). It can automate a wide range of activities, all within the context of your Kubernetes cluster. Here are some of the common CD actions Prow can handle:

1. Basic Deployments

Rolling Updates: Prow can orchestrate rolling updates to your applications, deploying new versions gradually while minimizing downtime. It can manage the rollout process, monitor the health of new pods, and roll back if issues are detected.
Blue/Green Deployments: Prow can facilitate blue/green deployments, where a new version (green) is deployed alongside the existing version (blue). Traffic is then switched to the green version once it’s verified, and the blue version is eventually decommissioned.
Canary Deployments: Prow can handle canary deployments, where a small portion of traffic is directed to the new version initially. This allows you to test the new version in a production environment with minimal risk before gradually increasing traffic.

2. Configuration Management

ConfigMap and Secret Updates: Prow can automate updates to ConfigMaps and Secrets in your Kubernetes cluster. This is useful for managing application configurations, environment variables, and sensitive data.

3. Image Updates

Triggering Deployments on New Images: Prow can monitor your container image registry and automatically trigger deployments when new images are available. This ensures that your applications are always running the latest versions.

4. Infrastructure Management

Kubernetes Resource Updates: Prow can apply changes to Kubernetes resources (Deployments, Services, Ingresses, etc.) based on your defined configurations. This allows you to manage your infrastructure as code and automate changes.

5. Advanced Deployment Strategies

A/B Testing: Prow can be used to set up A/B testing scenarios, where different versions of your application are deployed and traffic is split between them to compare their performance.
Feature Flags: Prow can integrate with feature flag management systems, allowing you to enable or disable features in your application dynamically without redeploying.

6. Self-Healing and Rollbacks

Automated Rollbacks: Prow can monitor the health of your deployments and automatically roll back to a previous version if issues are detected. This helps to minimize downtime and ensure service reliability.

7. Notifications and Reporting

Deployment Status Updates: Prow can provide notifications and reports on the status of your deployments, keeping you informed about the progress and any potential issues.

How Prow Achieves This

Prow leverages its tight integration with Kubernetes and its plugin system to perform these CD activities. You define your deployment processes in Prow job configurations, and Prow uses Kubernetes to execute those jobs and manage the deployments.

Plugins can further extend Prow’s CD capabilities by integrating with other tools and services, such as service meshes (Istio), monitoring systems, and feature flag platforms.