Kubernetes Authentication and Authorization using RBAC

15 min read5 days ago

Chapter # 1.0: Kubeconfig

1.1 What is Kubeconfig?

A kubeconfig file is a configuration file used by kubectl, the Kubernetes command-line tool, to access and manage Kubernetes clusters.

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority: /path/to/ca.crt  # Path to the CA certificate file
    server: https://<your-api-server-address>:6443
  name: my-cluster
users:
- name: my-user
  user:
    client-certificate: /path/to/client.crt  # Path to the client certificate file
    client-key: /path/to/client.key  # Path to the client key file
contexts:
- context:
    cluster: my-cluster
    user: my-user
    namespace: default
  name: my-context
current-context: my-context
preferences: {}

It essentially bundles together:

Cluster Information: Details about the Kubernetes API server (address, certificate authority).
User Credentials: Authentication information (certificates, tokens).
Contexts: A combination of a cluster and a user, allowing you to easily switch between different cluster access configurations.

1.2 Role of certificate-authority

1.2.1 Verification of API Server Identity

When kubectl connects to the API server, it needs to verify that it's talking to the legitimate server and not an imposter.
The API server presents its TLS certificate to kubectl.
kubectl uses the CA certificate specified in the certificate-authority field to verify the authenticity of the API server's certificate.
This ensures that kubectl is communicating with the correct server and not a malicious actor trying to intercept sensitive information.

Note: During K8 setup, same certificate is used to sign the API server certificate which is later used during authentication.

1.2.2 User Creation

In Kubernetes, users are often represented by their client certificates. These certificates are generated using tools like openssl.
The user creation process typically involves generating a private key and a Certificate Signing Request (CSR) for the user.

1.2.3 User Authentication

The CA comes into play when the user’s CSR is signed. This signature by the CA is what gives the user’s certificate its trustworthiness within the cluster.
The Kubernetes API server is configured to trust the CA. So, when a user presents their certificate, the API server checks if it’s signed by a trusted CA. If it is, the user is authenticated.

1.2.4 mTLS authentication

When a client (like kubectl) connects to the Kubernetes API server using client certificates, both the client and the server present their certificates for authentication.
This mutual authentication ensures that both parties are who they claim to be, establishing a strong trust relationship.

Chapter # 2.0: User

Kubernetes doesn’t have a built-in user management system. You’ll need to combine authentication and authorization methods to manage user access. Here’s a breakdown of the process:

2.1 Authentication

This step verifies the user’s identity. Common methods include:

2.1.1 X.509 Client Certificates

Generate a Certificate Authority (CA) and use it to sign client certificates for each user.
Configure the Kubernetes API server to trust this CA.
Users present their certificate when interacting with the cluster.

2.1.2 Service Account Tokens

Primarily used for applications within the cluster.
Each service account has an automatically generated token.
Can be used for user authentication but generally not recommended for direct user access.

2.1.3 OIDC (OpenID Connect)

Integrates with external identity providers (like Google, Azure AD, Okta).
Users authenticate with the provider, and Kubernetes validates the issued tokens.
Popular choice for enterprise environments.

2.1.4 Webhook Token Authentication

Use a custom webhook to authenticate tokens.
Offers flexibility for integrating with various authentication systems.

Note: In this post we will discuss to use x509 certificate based user creation.

2.2 Authorization

2.2.1 RBAC

Define Roles and ClusterRoles, which specify permissions (e.g., read pods, create deployments).
Create RoleBindings and ClusterRoleBindings to grant these roles to users or groups.

2.2.2 ABAC

An access request is made by a subject to perform an action on an object.
The ABAC system evaluates attributes of the subject, object, action, and environment against predefined policies.
If the attributes meet the conditions specified in the policies, access is granted; otherwise, it’s denied

2.2.3 Node authorization

Node authorization in Kubernetes is a specialized authorization mode designed to control the permissions of kubelets. Kubelets are agents that run on each node in a Kubernetes cluster and are responsible for managing the pods and containers on that node.

Note: In this post we will mainly focus on RBAC.

2.3 Create User

Following are steps to create user based on x509 certificate.

2.3.1 Generate a Private Key for the User

openssl genrsa -out user.key 2048

openssl genrsa: This invokes the OpenSSL command-line tool to generate an RSA private key.
-out user.key: This specifies the output file name for the private key (in this case, "user.key").
2048: This specifies the key size in bits. 2048 bits is a common and secure choice.

2.3.2 Create a Certificate Signing Request (CSR)

To produce the public signed certificate, we first need to create CSR as below.

openssl req -new -key user.key -out user.csr -subj "/CN=user,O=my-org,L=Fremont,ST=California,C=US"

openssl req -new: This invokes the OpenSSL command to create a new CSR.
-key user.key: This specifies the private key file to use when creating the CSR.
-out user.csr: This specifies the output file name for the CSR (in this case, "user.csr").
-subj "/CN=user,O=my-org,L=Fremont,ST=California,C=US": This sets the subject of the certificate.
/CN=user: Actual user name
/O=my-org: This is used to group user. During role binding, we can bind role to group and then later every user can simply keep joining this group. It helps to avoid submitting request for every user for role binding.
/L=Fremont: Locality (city)
/ST=California: State or Province
/C=US: Country

2.3.3 Public certificate

CSR is a request: The openssl req command creates a Certificate Signing Request (CSR). This CSR contains the user's public key along with other identifying information.

CA signs the CSR: The CSR is then sent to a Certificate Authority (CA). The CA verifies the information in the CSR and, if everything checks out, signs it with their private key. This signature creates the actual public certificate, which includes the user’s public key and the CA’s signature.

Authority

If you’re using a managed Kubernetes service (like EKS, AKS, GKE), the cloud provider usually acts as the CA and handles the certificate issuance process.
If you’re managing your own Kubernetes cluster (e.g., with kubeadm), you'll likely have your own CA setup. You would use tools like openssl to sign the CSR with your CA's private key.

Example with kubeadm

If you have a kubeadm cluster, you can sign the CSR using the cluster's CA like this:

openssl x509 -req -in user.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user.crt -days 365

2.4 Update kubeconfig file

The kubectl config set-credentials command is used to configure and manage user credentials within your kubeconfig file.

kubectl config set-credentials jane-doe --client-certificate=jane.crt --client-key=jane.key

<name>: This is a unique name you give to the credential entry. This name is used to reference the credentials in contexts.
--client-certificate: Path to the client public certificate file.
--client-key: Path to the client private key file.

2.5 Update kubectl context

The kubectl config set-context command is used to create or modify contexts within your kubeconfig file. Contexts in Kubernetes provide a way to group together cluster, user, and namespace information, making it easier to switch between different Kubernetes environments.

kubectl config set-context dev-cluster --cluster=my-cluster --user=jane-doe --namespace=development

<name>: This is a unique name for the context.
--cluster: The name of the cluster to associate with this context (must already be defined in your kubeconfig).
--user: The name of the user to authenticate as (must already be defined in your kubeconfig).
--namespace: The default namespace to use for this context.

2.6 Available context

The kubectl config get-contexts command is used to display the available contexts in your kubeconfig file. It shows you the configured contexts and helps you understand which clusters and users are associated with each context.

$ kubectl config get-contexts
CURRENT   NAME             CLUSTER          AUTHINFO    NAMESPACE
          docker-desktop   docker-desktop   docker-desktop   default
          minikube         minikube         minikube         default

2.7 Use context

kubectl config use-context <context-name> command is used to switch between different contexts in your kubeconfig file.

Chapter # 3.0: Role and RoleBinding

3.1 Role

A Role defines a set of permissions within a specific namespace. It specifies what actions can be performed on which resources.

# pod-reader-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development  # The namespace this Role applies to
  name: pod-reader 
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

3.2 RoleBinding

A RoleBinding grants the permissions defined in a Role to a specific user, group, or service account.

# pod-reader-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: development  # Must be in the same namespace as the Role
subjects:
- kind: User
  name: "CN=user,O=my-org,L=Fremont,ST=California,C=US"  # Replace with actual user's CN from their certificate
  apiGroup: rbac.authorization.k8s.io 
roleRef:
  kind: Role
  name: pod-reader  # Name of the Role to bind
  apiGroup: rbac.authorization.k8s.io

3.3 Who can apply these files?

Cluster Admin: By default, the cluster admin (the user with the cluster-admin ClusterRole) has full permissions to create, modify, and delete any resource in the cluster, including Roles and RoleBindings.
RBAC Authorization: Kubernetes uses Role-Based Access Control (RBAC) to determine who can do what. So, even a non-admin user can apply Role and RoleBinding YAML files if they have been granted the necessary permissions through RBAC.
Required Permissions: To apply a Role or RoleBinding, a user needs permissions to create those resources within the specified namespace. This usually involves verbs like create and update on the roles and rolebindings resources in the rbac.authorization.k8s.io API group.

Chapter # 4.0: ClusterRole and ClusterRoleBinding

ClusterRoles and ClusterRoleBindings in Kubernetes are similar to Roles and RoleBindings, but they operate at the cluster level instead of being confined to a specific namespace.

4.1 ClusterRole

A ClusterRole defines permissions across the entire cluster. It grants access to cluster-scoped resources (like nodes, persistent volumes) and namespace-scoped resources in all namespaces.
Use ClusterRoles when you need to define permissions that apply globally.

# cluster-monitoring-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-reader
rules:
- apiGroups: ["", "apps", "extensions", "metrics.k8s.io"]
  resources: ["pods", "nodes", "deployments", "replicasets", "daemonsets", "statefulsets", "jobs", "cronjobs", "pods/log", "nodes/metrics"]
  verbs: ["get", "list", "watch"]

4.2 ClusterRoleBinding

A ClusterRoleBinding grants the permissions defined in a ClusterRole to users, groups, or service accounts.
This allows you to assign cluster-wide permissions to specific subjects.

# cluster-monitoring-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: monitoring-reader-binding
subjects:
- kind: User
  name: "CN=monitoring-user,O=my-org"  # Replace with the actual user's CN
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: monitoring-reader  # Name of the ClusterRole to bind
  apiGroup: rbac.authorization.k8s.io

Chapter # 5.0: Service Account

A Service Account in Kubernetes is a special type of account used by applications running within the cluster to access the Kubernetes API. They provide an identity for pods and other processes to interact with the API server.

5.1 Why Service Accounts?

Application Identity: When an application needs to interact with the Kubernetes API (e.g., to read secrets, create resources), it needs an identity. Service Accounts provide this identity.
Security: Instead of hardcoding credentials into application code, Service Accounts allow you to manage permissions through RBAC, limiting the potential damage if an application is compromised.
Namespace Scoped: Service Accounts, like many Kubernetes resources, are namespace scoped, enabling you to control access within specific namespaces.

5.2 How to Create a Service Account?

You can create a Service Account using kubectl create serviceaccount or by defining it in a YAML file.

1. Using kubectl create:

kubectl create serviceaccount my-service-account -n my-namespace

2. Using a YAML file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: my-namespace

5.3 Role for service account

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: my-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

5.4 RoleBinding for service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: my-namespace
subjects:
- kind: ServiceAccount
  name: my-service-account # name of the Service account
  namespace: my-namespace # Namespace where service account exist.
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

5.5 Key points

Tokens: When a pod uses a Service Account, Kubernetes automatically mounts a token into the pod, which the application can use to authenticate with the API server.
Default Service Accounts: Each namespace has a default Service Account. Pods that don’t specify a Service Account will use this default.
Security: Follow the principle of least privilege. Grant only the necessary permissions to Service Accounts.

5.6 How to declare service account for pod?

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account  # Name of the Service Account to use
  containers:
  - name: my-container
    image: nginx:latest

5.7 How K8 store service account?

system:serviceaccount:<namespace>:<service_account> pattern is used to uniquely identify a service account in Kubernetes.

system:: This prefix indicates that the account is part of the Kubernetes system, rather than a user-created account.
serviceaccount:: This specifies that the account is a service account.
default:: This is the namespace where the service account resides. In this case, it's the default namespace.
<service_account>:: This is the name of the service account itself.

So, putting it all together, system:serviceaccount:default:test-sa refers to the service account named "test-sa" in the "default" namespace.

Why this pattern?

Uniqueness: This pattern ensures that every service account in the cluster has a unique identifier, even if they have the same name in different namespaces.
RBAC: This identifier is used in RoleBindings and ClusterRoleBindings to grant permissions to service accounts.
API Access: When a pod uses a service account, this identifier is included in the token that the pod uses to authenticate with the Kubernetes API server.

Example in RBAC:

If you want to grant a Role named “pod-reader” to the “test-sa” service account, you would use the following in a RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-binding
subjects:
- kind: ServiceAccount
  name: test-sa
  namespace: default # The namespace the service account is in
roleRef:
  #...

When to use full qualified in role binding?

When using a ClusterRoleBinding, which grants permissions cluster-wide, you might need to specify the full service account name to avoid ambiguity if there are service accounts with the same name in different namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-cluster-role-binding
subjects:
- kind: ServiceAccount
  name: system:serviceaccount:my-namespace:my-service-account  # Full qualified name
roleRef:
  kind: ClusterRole
  name: my-cluster-role 
  apiGroup: rbac.authorization.k8s.io

This ensures that the ClusterRoleBinding specifically targets the service account with that name in the my-namespace namespace, even if there are other service accounts with the same name in different namespaces.

5.8 How Kubernetes manages Service account?

When Kubernetes creates a service account, it does not internally use x509 certificates. Instead, it uses JSON Web Tokens (JWTs) to authenticate service accounts.

Here’s how it works:

Token Generation: When you create a service account, Kubernetes automatically generates a JWT for that service account. This token contains information about the service account, such as its name, namespace, and assigned permissions.
Token Storage: The token is stored as a secret within the cluster.
Token Mounting: When a pod uses a service account, Kubernetes automatically mounts the token into the pod at a specific location (usually /var/run/secrets/kubernetes.io/serviceaccount).
API Authentication: The application within the pod can then use this token to authenticate with the Kubernetes API server. The API server validates the token and authorizes the request based on the service account’s permissions.

Why JWTs instead of x509 certificates?

Simplicity: JWTs are simpler to manage and distribute compared to x509 certificates.
Automation: Kubernetes automates the generation and mounting of service account tokens, making it easier for applications to use.
Flexibility: JWTs can be easily used for various authentication and authorization scenarios within the cluster.

x509 certificates are primarily used for:

User authentication: When human users or external entities need to access the cluster.
Kubelet authentication: Kubelets use x509 certificates to authenticate with the API server.

Chapter # 6.0: Group

Instead of managing permissions for individual users, you can assign permissions to a group, and all members of that group automatically inherit those permissions.

6.1 Define Group in Certificate

When generating x509 certificates for users, include the O (Organization) or OU (Organizational Unit) field in the certificate's Subject to represent the group.

openssl req -new -key user1.key -out user1.csr -subj "/CN=user1,O=my-group,OU=dev-team"

CN: user1 (the user's individual identifier)
O: my-group (the group the user belongs to)
OU: dev-team (an optional sub-group or team)

6.2 Reference Group in RoleBinding

In your RoleBinding or ClusterRoleBinding, use the O or OU field to specify the group.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-group-rolebinding
  namespace: my-namespace
subjects:
- kind: Group
  name: my-group  # Only the group is included here
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: my-group-role
  apiGroup: rbac.authorization.k8s.io

6.3 Adding Multiple Users to the Group

To add more users to the group, simply generate their certificates with the same O or OU field value.

openssl req -new -key user2.key -out user2.csr -subj "/CN=user2,O=my-group,OU=dev-team"

Now, both user1 and user2 will be considered part of the "my-group" group and will be granted the permissions defined in the RoleBinding.

6.4 Refer OU in RoleBinding

If you’ve used the OU field to define teams or sub-groups within a larger organization, referencing it in RoleBindings allows you to grant more specific permissions to those sub-groups.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-ou-rolebinding
subjects:
- kind: Group
  name: dev-team  # Matches the 'OU' field in the certificate
  apiGroup: rbac.authorization.k8s.io
roleRef:
  #...

Chapter # 7.1 Role aggregation

Role aggregation allows you to combine multiple roles into a single aggregated role. This simplifies RBAC management by allowing you to define smaller, more focused roles and then combine them as needed .

To aggregate roles, you use the aggregationRule field in a ClusterRole. This field allows you to specify a label selector that will match other ClusterRoles. Kubernetes will then automatically include the rules from those matched ClusterRoles in the aggregated ClusterRole .

For example, you could create separate roles for managing deployments, services, and pods, each with its own set of permissions. Then, you could create an aggregated role that combines these roles, granting a user or service account all the permissions needed to manage those resources.

Here’s an example of how to define a ClusterRole that aggregates permissions from other Roles:

7.1 Define the Roles

First, create the Roles that you want to aggregate. These Roles can be in different namespaces or even cluster-scoped.

# role1.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: namespace1
  name: role1
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

---

# role2.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: namespace2
  name: role2
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list"]

7.2 Define the Aggregated ClusterRole

Next, create a ClusterRole that aggregates the permissions from the Roles you defined. Use the aggregationRule to specify which Roles to include.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: aggregated-role
aggregationRule:
  clusterRoleSelectors:  # Select ClusterRoles to aggregate
  - matchLabels:
      rbac.example.com/aggregate-to-monitoring: "true"  # Select Roles with this label
rules:  # Initially empty, rules will be aggregated

7.3 Label the Roles

Add the label specified in the aggregationRule to the Roles you want to aggregate.

# role1.yaml (modified)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: namespace1
  name: role1
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"  # Add the label for aggregation
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

---

# role2.yaml (modified)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: namespace2
  name: role2
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"  # Add the label for aggregation
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list"]

7.4 Result

Now, the aggregated-role ClusterRole will have the combined permissions from role1 and role2. You can use this ClusterRole in ClusterRoleBindings to grant those aggregated permissions to users or service accounts.

7.5 Why only applicable to ClusterRole?

Role aggregation in Kubernetes, using the aggregationRule, is only available for ClusterRoles. You cannot aggregate regular Roles.

Here’s why this makes sense in the Kubernetes architecture:

Cluster-wide Scope: ClusterRoles have cluster-wide scope, meaning they can grant permissions to resources across all namespaces. This makes them suitable for aggregating permissions from multiple Roles that might exist in different namespaces.
Namespace Limitation: Roles, on the other hand, are namespace-scoped. They only grant permissions within a specific namespace. Aggregating namespace-scoped Roles wouldn’t be as useful, as the aggregated permissions would still be limited to a single namespace.
Centralized Management: ClusterRoles provide a central place to define and manage aggregated permissions, which can then be granted to users or service accounts across the cluster.

Chapter #8: Verify permission

The kubectl auth can-i command is a powerful tool in Kubernetes that allows you to check whether a user or service account has the necessary permissions to perform a specific action on a resource. This helps you understand and verify RBAC (Role-Based Access Control) policies in your cluster.

Here’s a breakdown of the command and related concepts:

8.1 `kubectl auth can-i` Command

Purpose: Checks if the current user or a specified user/service account can perform a given verb (action) on a resource.

Syntax:

kubectl auth can-i <verb> <resource> [options]

<verb>: The action you want to check (e.g., get, list, create, update, delete).
<resource>: The Kubernetes resource (e.g., pods, deployments, services).
-n <namespace>: Specifies the namespace to check permissions in.
-A: Checks permissions in all namespaces.
--as=<user>: Checks permissions as the specified user or service account.
--list: Lists all allowed verbs for the given resource.

8.2 Examples

Check if the current user can create pods in the default namespace:

kubectl auth can-i create pods

Check if the service account my-sa in the my-namespace namespace can list deployments:

kubectl auth can-i list deployments -n my-namespace --as=system:serviceaccount:my-namespace:my-sa

List all allowed verbs for pods in all namespaces for the current user:

kubectl auth can-i --list pods -A

Specify api group:

kubectl auth can-i list deployments --api-group apps

This post is based on interaction with https://gemini.google.com/.

Enjoy learning :-)