CLI Best Practices
Node-level access is the most privileged thing most platform engineers do routinely.
//
Search across all documentation pages
Node-level access is the most privileged thing most platform engineers do routinely.
These practices keep node debugging fast, repeatable, and safe - so that dropping below the CRI is a deliberate act rather than a habit.
Treat A and B as team policy: they define when node access is warranted and how you get it.
Treat C through E as craft: how you run commands once you are on the box, and how you turn a good session into a script nobody has to reinvent at 3am.
Adopt the scripting practices in E first if you only have time for one group. Codified checks are what stop tribal knowledge from evaporating.
kubectl describe pod, kubectl get events --sort-by=.lastTimestamp, and your logging backend answer most questions. Node access is for when they are empty or contradict reality.ContainerCreating that never resolves, ImagePullBackOff with no detail, DiskPressure, throttling invisible in metrics, or a NotReady node.kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> shares the target's namespaces, is RBAC-gated, and leaves an audit trail.kubectl debug node/<name> instead of SSH for host work. It mounts the host root at /host and works in clusters where SSH is deliberately disabled.kubectl cordon first, kubectl drain --ignore-daemonsets if you intend to be invasive.crictl for anything the kubelet knows about. Pods, containers, images, and logs at the CRI layer. It is the modern replacement for docker ps on a node.ctr only below CRI. Snapshot usage, content store, and namespace inspection. Always pass -n k8s.io or you will see nothing.nsenter when the image has no shell. Enter only the namespaces you need, and omit -m when you want to run host binaries like ss or tcpdump against the container's network.memory.events, cpu.stat, and memory.current under /sys/fs/cgroup are the kernel's own numbers, not a scraped approximation.journalctl -u containerd -u kubelet for runtime-level failures. CNI errors and snapshotter problems appear there and nowhere else.crictl ps, inspect, logs, and stats are safe on production. crictl rm, rmi, and stop fight the kubelet and cause confusing reconciliation.ctr run on a live node. The kubelet does not account for them, and they escape eviction and resource limits./var/lib/kubelet or /var/lib/containerd. You will corrupt state the runtime owns. Fix the declarative source instead.tcpdump -c 200 or -G 60 -W 1; an unbounded capture fills the node's disk and triggers evictions./tmp disappears with the next node rotation.-o json and pipe through jq. crictl inspect <id> | jq '.status.exitCode' is greppable, diffable, and pasteable into an incident channel.top -b -n1 and iostat -x 1 3 produce a snapshot you can attach to a ticket; a live TUI produces nothing.crictl inspect <id> | jq '.info.config.labels' gives io.kubernetes.pod.name and io.kubernetes.pod.namespace.date -u at the start of a session and journalctl --since with explicit UTC times so your evidence lines up with dashboards.kubectl cp it, so every engineer runs the same checks.set -euo pipefail in every script. Silent failures during an incident cost more time than the script saves.cpu.stat check three times, it belongs in Prometheus as container_cpu_cfs_throttled_seconds_total, not in your shell history.A minimal starting point that covers the common node failures.
#!/usr/bin/env bash
# node-triage.sh - read-only node health snapshot
set -euo pipefail
echo "== when =="; date -u
echo "== disk =="
df -h /var/lib/containerd /var/lib/kubelet /
echo "== memory + load =="
free -m
uptime
echo "== services =="
systemctl is-active kubelet containerd || true
echo "== runtime errors (last 15m) =="
journalctl -u containerd -u kubelet --since "15 min ago" -p err --no-pager | tail -30
echo "== not-ready sandboxes =="
crictl pods --state NotReady
echo "== recently exited containers =="
crictl ps -a --state Exited | head -15
echo "== kernel oom kills =="
journalctl -k --since "1 hour ago" --no-pager | grep -i "oom-kill" || echo "none"pods/ephemeralcontainers and nodes/proxy are grantable, revocable, and logged.kubectl debug node/... pods persist until you delete them.Confirm you left the node exactly as you found it - no stray containers, no running captures, no edited runtime state.
Uncordon anything you cordoned, and delete any debug pod you created.
Then ask the durable question: what alert, limit, probe, or dashboard would have answered this without a shell? Write that down before the incident closes.
Should engineers have SSH to production nodes at all?
Increasingly no. kubectl debug node/<name> plus ephemeral containers cover nearly every case with better auditing.
Is crictl safe to run during an incident?
The read commands are. Avoid rm, rmi, and stop, which conflict with the kubelet's reconciliation loop.
Why does ctr show nothing on my node?
You omitted -n k8s.io. containerd namespaces are mandatory, and Kubernetes workloads live in that one.
Where should triage scripts live? In the platform repo, reviewed like any code, and baked into your debug image so the version on the node matches the version in git.
How do I debug a distroless image with no shell?
Attach an ephemeral container with --target, or use nsenter from the host with only -n or -p so you keep host binaries.
What is the single highest-leverage practice here? Codify the checks. A read-only triage script in the repo outlives every individual's memory of what to look at.
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).
Reviewed by Chris St. John·Last updated Jul 16, 2026