Don’t have time to read this post? I get it. The answer to your question is /var/lib/kubelet/<pod-id>/volumes/.

Every so often I will be pairing with someone or showing off a demo and realize that some common operation I perform is not well-documented. We all have examples of domain knowledge we learned somewhere along the way and stashed in our toolbox. Whenever I encounter these situations I try to remember to post about them so that the next person can find the information at least a little bit faster than I did.

Recently, I was troubleshooting an issue where some files were being mounted as a volume to a container in a Pod on a Kubernetes cluster. The volume appeared to be mounted successfully, but the container was failing to authenticate with credentials that should have been present in the mounted directory. I wanted to validate that the credentials were in fact valid, but this particular container was using a minimal OCI image (shout out distroless 👏), so using kubectl exec to get a shell was not possible.

At this point I fell back to the familiar process of accessing the Node to see what was going on. Kubernetes can sometimes feel like an opaque API, but it’s helpful to remember that behind the scenes everything is still processes and files on physical hardware with varying levels of isolation. If we quickly spin up a kind cluster, we can start to explore a node that itself is a container running on our local machine.

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

By default, we will have a single node that is serving as both the control plane and a target for scheduling workloads.

$ docker container ls
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS                       NAMES
f8a584b70c83   kindest/node:v1.25.3   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   127.0.0.1:46125->6443/tcp   kind-control-plane

We can use a Secret volume mount as an example (acknowledging that if you could read the Secret then you wouldn’t need to be checking the volume contents). We’ll mount it onto a simple container image that is built with gcr.io/distroless/cc as the base, and only runs a single binary that sleeps forever.

$ kubectl create secret generic super-secret --from-literal=test=data
secret/super-secret created

apiVersion: v1
kind: Pod
metadata:
  name: sleep
spec:
  containers:
  - name: sleep
    image: hasheddan/sleep:v0.0.1
    volumeMounts:
    - name: secrets
      mountPath: /data/secret
  volumes:
  - name: secrets
    secret:
      secretName: super-secret

$ kubectl apply -f pod.yaml 
pod/sleep created

$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
sleep   1/1     Running   0          5s

We could use docker exec to get a shell into the kind node (if this was a cloud-hosted VM you could use ssh), but in the spirit of “not being able to get a shell” let’s do it all on our host machine.

Step 1: Finding the Node Filesystem Link to heading

As mentioned previously, our node is just a container running on our local machine when we use kind. Before we can look at data in our container running in kind, we first need to find the filesystem of the node itself. We’ll avoid getting too deep into how overlayfs works, and instead just access the filesystem of the node via the proc filesystem. In order to do so, we need to get the PID of the node container, which Docker makes possible with a single command.

$ docker container inspect kind-control-plane | jq .[0].State.Pid
958956

With PID in hand, we can access the root directory of the node.

Note: you will likely need elevated privileges to access the root filesystem.

$ ls /proc/958956/root
bin  boot  dev  etc  home  kind  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

This mirrors what we would see if we used docker exec instead.

$ docker exec -it kind-control-plane ls
bin  boot  dev	etc  home  kind  lib  lib32  lib64  libx32  media  mnt	opt  proc  root  run  sbin  srv  sys  tmp  usr	var

Step 2: Get the Pod UID Link to heading

Next, we need to know the UID of the Pod for the container we are interested in.

$ kubectl get pod sleep -o=jsonpath={.metadata.uid}
71b73425-1838-4688-9255-57fbebaa6d43

The kubelet writes data to the /var/lib/kubelet directory, so it is a safe bet to look there for mounted volumes.

Step 3: Finding the Volume Mount Link to heading

The /pods subdirectory has data for each Pod that has been scheduled to this node, which in our case is all Pods.

$ ls /proc/958956/root/var/lib/kubelet/pods
0d94b24ab949b9eb0e6237e4515cb37b      305850cc48c365264e55686e68437f03      57b07309-36c0-4a02-abec-5b97196bac0f  71b73425-1838-4688-9255-57fbebaa6d43  a22bc248-652a-4142-b3a3-203885aa6df1
2ff98227-a55e-4907-b666-2cb1128e6ad8  3401ce1b-9970-4139-bcf0-400ec0ebca9e  6d3dda2cad9846e0d48dbd5d5b9f59fc      9396d9ae-116a-434d-b0c2-d56df3f7d663  aaff90ec64f346d418f0a93d766752c5

Pods are identified by their UID, and within each Pod directory, there is a dedicated volumes subdirectory that is segmented by type.

$ ls /proc/958956/root/var/lib/kubelet/pods/71b73425-1838-4688-9255-57fbebaa6d43/volumes/
kubernetes.io~projected  kubernetes.io~secret

For our simple example, we have both a projected token volume for our ServiceAccount, as well as the Secret volume we explicitly requested. Looking inside the Secret volume shows a test file that matches our key, and the contents include the value.

$ cat /proc/958956/root/var/lib/kubelet/pods/71b73425-1838-4688-9255-57fbebaa6d43/volumes/kubernetes.io~secret/secrets/test
data

Bonus Round: proc Inception Link to heading

If we instead wanted to look at the filesystem of our sleep container directly, we could follow the exact same steps in the kind node as we did to access its filesystem. Importantly, we need to find the PID of the sleep process in the kind node, which will be different than the one we observe on the host.

$ ls -l /proc/958956/root/proc/*/exe | grep sleep
lrwxrwxrwx 1 root  root  0 Mar 18 14:55 proc/2137/exe -> /sleep

We can then access the root filesystem of the sleep process, and find the data that the kubelet mounted on our behalf.

$ cat /proc/958956/root/proc/2137/root/data/secret/test
data

Closing Thoughts Link to heading

I hope this post has helped at least one person fix an issue a little faster than they would have without it. Always remember that all data is just bits somewhere in the machine!

If you have any questions, thoughts, or suggestions, please feel free to send me a message @hasheddan on Twitter or @hasheddan@types.pl on Mastodon!