Why Platform Work Drops to the Node
Kubernetes is an abstraction over Linux hosts. When the abstraction leaks, you debug the host.
Summary
- Definition: "Dropping to the node" means leaving
kubectland working directly on a worker's Linux host, its container runtime, and the kernel primitives that back pods. - Insight:
kubectltalks to the API server, but the API server only knows what the kubelet reports. When the kubelet, containerd, runc, or the kernel is the problem, the control plane has no view into it. - Key Concepts: the CRI boundary (kubelet to containerd), runc and OCI runtimes, cgroups v2 and namespaces, and node-level tooling (
crictl,ctr,nsenter,journalctl). - When to Use: a pod is stuck
ContainerCreating, images will not pull, a node isNotReady, throttling or OOM kills have no obvious cause, or the API is simply unreachable. - Limitations/Trade-offs: node access is powerful and dangerous. It bypasses admission control, audit trails, and Pod Security Standards, so it is a break-glass activity, not routine.
- Related Topics: the CRI, containerd, cgroups v2, Linux namespaces, and kubelet internals.
Foundations
A Kubernetes cluster is a set of Linux machines coordinated by a control plane.
Every pod you schedule ultimately becomes processes on some worker node, isolated by kernel namespaces and constrained by cgroups.
The mental model is a stack of layers. From top to bottom: your manifest, the API server, the scheduler, the kubelet on a node, the container runtime (containerd), the OCI runtime (runc), and finally the Linux kernel.
kubectl operates at the top of that stack. It reads and writes objects in etcd through the API server.
Most days that is enough, because the layers below faithfully do what the objects describe.
Platform work starts where that faithfulness breaks down.
When the kubelet cannot start a sandbox, or containerd cannot unpack an image, or the kernel kills a process for exceeding a memory cgroup, the failure happens below the API server's line of sight.
You cannot fix what you cannot see, so you descend to the layer where the truth lives.
Mechanics & Interactions
The key seam is the Container Runtime Interface (CRI), a gRPC contract between the kubelet and the runtime.
On modern clusters the runtime is containerd, reached over a Unix socket at /run/containerd/containerd.sock. Docker Engine is not in this path on nodes, and dockershim was removed in Kubernetes 1.24.
The kubelet asks containerd to create a pod sandbox (the pause container that holds the shared network and IPC namespaces), then to create and start each app container inside it.
containerd in turn calls runc, an OCI runtime, to actually clone() new namespaces, mount the root filesystem, apply the cgroup, and exec your entrypoint.
This is why node debugging needs its own tools.
kubectl get pods shows you the object's status as reported upward. crictl ps shows you what containerd actually has running, which can differ during a failure.
$ crictl ps -a
CONTAINER IMAGE STATE NAME ATTEMPT
7f3a1b2c9d0e nginx@sha256 Running web 0
2b8c4d1a6f5e nginx@sha256 Exited web 3The ATTEMPT counter and repeated Exited states expose a crash loop that kubectl summarizes but does not fully explain.
Below containerd, the kernel enforces isolation with namespaces (what a process can see) and cgroups v2 (what a process can use).
A container is not a first-class kernel object. It is a normal process with its own set of namespaces and a cgroup, which is exactly why nsenter and the cgroup filesystem let you inspect it from the host.
nsenter enters a running container's namespaces from the host, giving you a shell inside the container even when it has no shell of its own and kubectl exec is failing.
The cgroup v2 filesystem under /sys/fs/cgroup exposes real-time resource accounting: memory.current, memory.max, and memory.events tell you whether the kernel is throttling or killing your workload.
Advanced Considerations & Applications
Node access is a security boundary you are deliberately crossing.
Anything you do with ctr or crictl skips the API server, so there is no admission webhook, no RBAC check, and no audit event in the cluster log.
Treat it as break-glass: access should be logged at the infrastructure layer (SSH, SSM Session Manager, or a privileged debug pod), time-boxed, and reviewed.
Prefer the least-privileged tool that answers the question.
crictl is CRI-aware and safe for read-only inspection of Kubernetes-managed containers. ctr is containerd's low-level client and can touch containerd's internal namespaces, including k8s.io, where it can break running pods if you are careless.
On managed platforms you often cannot SSH at all.
There the idiomatic path is kubectl debug node/<name>, which schedules a privileged pod in the host namespaces, or an ephemeral debug container attached to a target pod with kubectl debug -it <pod> --image=busybox --target=<container>.
These give you node-level reach through the API server, preserving the audit trail while still crossing into the host's view.
Common triage that requires the node includes image pull failures (registry auth or disk pressure visible only to containerd), sandbox creation errors (CNI plugin failures the kubelet logs but does not surface), and resource pressure (OOM kills and CPU throttling recorded in cgroup counters and dmesg).
The payoff is decisiveness. Instead of guessing from a kubectl describe event, you read the runtime and kernel state directly and know.
Common Misconceptions
"Docker runs my pods, so I should use docker on the node."
No. containerd runs pods via the CRI. docker ps on a node either fails or shows nothing relevant. Use crictl.
"kubectl exec and node access are the same thing."
kubectl exec runs a process inside a container through the kubelet. It fails when the kubelet or runtime is unhealthy, which is exactly when node tools like nsenter become necessary.
"If kubectl get nodes says Ready, the node is fine."
Ready reflects the kubelet's self-report. A node can be Ready while suffering disk pressure, PID exhaustion, or a wedged containerd that has not yet failed its health probe.
"I need root on the node for everything."
Much read-only inspection works through kubectl debug and a privileged ephemeral container, no SSH required, which is preferable on managed clusters.
"Node debugging replaces observability." It does not. Metrics, logs, and traces should catch most issues first. Node work is the last resort when telemetry runs out.
FAQs
When should I drop to the node instead of using kubectl?
When the failure is below the API server: stuck sandboxes, image pull errors, NotReady nodes, unexplained OOM or throttling, or an unreachable API.
Is Docker installed on my Kubernetes nodes? Usually not. containerd is the CRI runtime. Docker may exist on build machines, but it is not in the pod runtime path.
What is the safest tool to start with?
crictl, in read-only mode (ps, inspect, logs). It is CRI-aware and will not disturb containerd's internal state the way careless ctr use can.
Can I avoid SSH on managed nodes?
Yes. Use kubectl debug node/<name> or an ephemeral debug container. Both reach node-level state through the API server with an audit trail.
Why can crictl ps and kubectl get pods disagree?
kubectl shows the object's reported status; crictl shows containerd's actual container state. They diverge precisely during the failures you are debugging.
Does node access bypass RBAC and Pod Security Standards? Yes. Direct runtime access has no admission or RBAC checks, so it must be treated as privileged break-glass.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).