Github Repo found here: https://github.com/heilancoos/ludus_k8s
Introduction
In my blog on Attacking and Defending Kubernetes, I covered several different attack techniques and provided Falco detections for them. While writing that blog over the course of 3 months, I found that I spent most of my time simply trying to get things working properly as things tend to go with security research.
There is something to be said about the fact that setting up and configuring environments is worthwhile for upskilling and gaining a fuller understanding of different tools. However, at the same time, it can be a significant barrier to entry when it comes to security research. This problem is the underlying philosophy why Bad Sector Labs released their cyber range deployment tool, Ludus. Ludus allows users to quickly and easily deploy environments in Proxmox using Ansible and Packer. This has made security research infinitely easier and I wish I was aware of it earlier.
I’ve decided to release a collection of Ansible roles that makes setting up Kubernetes environments a lot easier for security research. This blog post will cover the main features of ludus_k8s.
Architecture Overview
ludus_k8s supports microk8s and kubeadm for cluster deployment methods. The default is microk8s, but the other options can be set by configuring k8s_flavor in the main configuration file.
Falco is used for runtime detections. It watches node activity and alerts based on pre-defined or custom rules. I talk more in-depth about Falco in my other blog here. Falcosidekick then forwards those alerts to a configured endpoint. In ludus_k8s, there is built-in support for running Falcosidekick UI to display alerts. There is also support for Grafana/Loki stack, where Loki serves as centralized log aggregation and then Grafana visualizes those logs in a dashboard.
Additionally, for users with the use case of deploying several nodes within a single cluster, ludus_k8s supports multinode functionality for both deployment methods.
Finally, since the whole reasoning behind this is for security research, ludus_k8s provides an easy way to configure common vulnerable misconfigurations such as unauthenticated access to the API server and Kubelet API, ETCD service exposure, insecure RBAC configurations, and more.
Core K8s features
The install_k8s role handles features like cluster configuration and embedding vulnerabilities. microk8s is the default flavor.
Pods
custom_pods takes a list where each entry is either an inline pod manifest or a path string to a YAML file.
custom_pods:
# Inline pod sepc
- apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
# File path
- "/home/localuser/my-deployment.yaml"
RBAC
Service accounts, namespaces, Roles/ClusterRoles, and RoleBindings/ClusterRolebindings can be configured with rbac_namespaces, rbac_service_accounts, rbac_roles, and rbac_bindings.
rbac_namespaces:
- dev
rbac_service_accounts:
- name: app-sa
namespace: dev
rbac_roles:
- kind: ClusterRole
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
rbac_bindings:
- kind: ClusterRoleBinding
name: secret-reader-binding
role_ref:
kind: ClusterRole
name: secret-reader
subjects:
- kind: ServiceAccount
name: app-sa
namespace: dev
Network Policies
Specific Network Policies Providers can be enabled with k8s_cni for kubeadm deployments. Currently supported providers are Calico and Cilium.
For microk8s, Calico can be enabled with enable_network_policies.
Pod Security Admission (PSA)
PSA is an Admission Controller that can enforce Pod Security Standards which define different isolation levels in Pods. It is enabled with enable_pod_security_admission and defined with psa_namespaces:
psa_namespaces:
- name: prod
enforce: restricted
audit: restricted
warn: restricted
- name: staging
enforce: baseline
Admission Controllers
enable_admission_controller will deploy a Kyverno admission controller that is either Validating or Mutating. There are four pre-defined policies: block_privileged, require_labels, inject_security_context, and add_default_labels.
block_privileged: A validating admission controller that refuses the deployment of any privileged containers and hostPath mounts.require_labels: A validating admission controller that requires environment and app labels on namespaces and pods.inject_security_context: A mutating admission controller that injects a hardened securityContext into pods.add_default_labels: A mutating admission controller that adds an environment labels to namespaces without it.
Custom policies can also be passed through kyverno_custom_policies.
Multinode
For simple environments, often a singular node suffices. For more complex environments, a multi-node cluster can be used to ensure high-availability. A multi-node cluster consists of one Master node and one or more Worker nodes.
The Master node is where the Control Plane lives and handles things like scheduling workloads. Worker nodes are where the workloads actually get run.
Note: On microk8s clusters with multiple nodes, High-Availability is managed by Dqlite rather than ETCD. Consequentially, unauth_etcd is not available on microk8s clusters with multinode: true.
Headlamp Dashboard
Since the Kubernetes Dashboard is deprecated, Headlamp serves as an alternative.
enable_headlamp deploys the Headlamp Dashboard and specific configurations such as headlamp_namespace and headlamp_nodeport can be defined as well.
Telemetry
Falco is installed by default relying on both syscalls and k8saudit for detections. UI selection is defined by ui and supports either grafana or falcosidekick which deploys the FalcoSidekick web ui.
There are some pre-defined Falco rules that live in roles/telemetry/tasks/deps/rules.yaml for easy reference. falco_custom_rules takes a string of rule names to deploy with Helm.
Example:
falco_custom_rules:
- "Anonymous Request Allowed"
Insecure Configurations
Unauthenticated API Access
enable_unauth_api and enable_unauth_kubelet can be enabled to configure unauthenticated access to the API server and the Kubelet API. Additionally, Kubelet API is configured with the authorization mode set to AlwaysAllow.
enable_unauth_api does not change the authorization mode however. anon_cluster_admin: true will bind cluster-admin to system:anonymous.
Unauthenticated ETCD Access
unauth_etcd: true exposes ETCD without authentication and installs etcdctl and kubetcd.
Anonymous Cluster Admin
anon_cluster_admin binds cluster-admin to system:anonymous
Kubernetes Remote Code Execution Via Nodes/Proxy GET Permission
A technique first detailed by Graham Helton, gh_nodes_proxy enables a demo environment that sets up a ClusterRole with the permissions nodes/proxy get.
Details on how the technique works can be found here.
Pod Remote Code Execution
demo_pod_rce: true deploys a Flask app with a command injection vulnerability.
Final Thoughts
It is my hope that this role collection will be helpful to security researchers, sysadmins, and anyone wanting to dive further into Kubernetes. For future expansion plans, I hope to add support for other Kubernetes deployment methods and additional features to make deploying Kubernetes clusters even more accessible.
Copy Link