CLI Best Practices

Node-level access is the most privileged thing most platform engineers do routinely.

These practices keep node debugging fast, repeatable, and safe - so that dropping below the CRI is a deliberate act rather than a habit.

How to Use This List

Treat A and B as team policy: they define when node access is warranted and how you get it.

Treat C through E as craft: how you run commands once you are on the box, and how you turn a good session into a script nobody has to reinvent at 3am.

Adopt the scripting practices in E first if you only have time for one group. Codified checks are what stop tribal knowledge from evaporating.

A - Know When to Drop to the Node

Exhaust the control plane first. kubectl describe pod, kubectl get events --sort-by=.lastTimestamp, and your logging backend answer most questions. Node access is for when they are empty or contradict reality.
Drop down when the symptom is below the pod. ContainerCreating that never resolves, ImagePullBackOff with no detail, DiskPressure, throttling invisible in metrics, or a NotReady node.
Prefer ephemeral containers over SSH. kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> shares the target's namespaces, is RBAC-gated, and leaves an audit trail.
Use kubectl debug node/<name> instead of SSH for host work. It mounts the host root at /host and works in clusters where SSH is deliberately disabled.
Never debug a node that is still taking traffic if the fix is disruptive. kubectl cordon first, kubectl drain --ignore-daemonsets if you intend to be invasive.

B - Use the Right Tool for the Layer

crictl for anything the kubelet knows about. Pods, containers, images, and logs at the CRI layer. It is the modern replacement for docker ps on a node.
ctr only below CRI. Snapshot usage, content store, and namespace inspection. Always pass -n k8s.io or you will see nothing.
Never expect Docker Engine on a node. dockershim was removed in Kubernetes 1.24; containerd runs your pods. Docker Engine 29.6.1 is your build and Compose tool, not the runtime.
nsenter when the image has no shell. Enter only the namespaces you need, and omit -m when you want to run host binaries like ss or tcpdump against the container's network.
The cgroup v2 filesystem for ground truth. memory.events, cpu.stat, and memory.current under /sys/fs/cgroup are the kernel's own numbers, not a scraped approximation.
journalctl -u containerd -u kubelet for runtime-level failures. CNI errors and snapshotter problems appear there and nowhere else.

C - Read Before You Write

Default to read-only commands. crictl ps, inspect, logs, and stats are safe on production. crictl rm, rmi, and stop fight the kubelet and cause confusing reconciliation.
Never create containers with ctr run on a live node. The kubelet does not account for them, and they escape eviction and resource limits.
Do not hand-edit files under /var/lib/kubelet or /var/lib/containerd. You will corrupt state the runtime owns. Fix the declarative source instead.
Bound every capture. tcpdump -c 200 or -G 60 -W 1; an unbounded capture fills the node's disk and triggers evictions.
Copy artifacts off the node immediately. Anything you leave in /tmp disappears with the next node rotation.

D - Make Output Machine-Readable

Use -o json and pipe through jq. crictl inspect <id> | jq '.status.exitCode' is greppable, diffable, and pasteable into an incident channel.
Prefer batch flags over interactive tools. top -b -n1 and iostat -x 1 3 produce a snapshot you can attach to a ticket; a live TUI produces nothing.
Map containers back to Kubernetes with labels. crictl inspect <id> | jq '.info.config.labels' gives io.kubernetes.pod.name and io.kubernetes.pod.namespace.
Timestamp everything. Use date -u at the start of a session and journalctl --since with explicit UTC times so your evidence lines up with dashboards.
Capture the command, not just the output. A pasted result nobody can reproduce is worth less than the one-liner that produced it.

E - Script the Checks You Repeat

Write a single node triage script and put it in the repo. Disk, memory, throttling, kubelet and containerd status, top pods - in that order, in one run.
Make scripts read-only and idempotent. A triage script that mutates state cannot be handed to an on-call engineer under pressure.
Ship the script to the node rather than typing it. Bake it into the debug image, or kubectl cp it, so every engineer runs the same checks.
Set set -euo pipefail in every script. Silent failures during an incident cost more time than the script saves.
Turn every repeated manual check into an alert. If you have run the same cpu.stat check three times, it belongs in Prometheus as container_cpu_cfs_throttled_seconds_total, not in your shell history.
Keep the script portable across nodes. Resolve paths and PIDs at runtime rather than hardcoding cgroup paths that vary by QoS class.

A minimal starting point that covers the common node failures.

#!/usr/bin/env bash
# node-triage.sh - read-only node health snapshot
set -euo pipefail
 
echo "== when =="; date -u
 
echo "== disk =="
df -h /var/lib/containerd /var/lib/kubelet /
 
echo "== memory + load =="
free -m
uptime
 
echo "== services =="
systemctl

F - Keep Access Auditable and Scoped

Gate node access behind RBAC, not shared keys. pods/ephemeralcontainers and nodes/proxy are grantable, revocable, and logged.
Log the session. Cloud session managers record commands; SSH usually does not. Prefer the one that gives you an after-action record.
Never leave a privileged debug pod running. Ephemeral containers die with the pod, but kubectl debug node/... pods persist until you delete them.
Feed findings back into declarative config. A node fix that lives only in a shell session will be undone by the next node rotation.
Treat repeated node access as a signal. If the same class of incident sends you to the node monthly, the missing artifact is an alert, a probe, or a limit - not better shell skills.

When You Are Done

Confirm you left the node exactly as you found it - no stray containers, no running captures, no edited runtime state.

Uncordon anything you cordoned, and delete any debug pod you created.

Then ask the durable question: what alert, limit, probe, or dashboard would have answered this without a shell? Write that down before the incident closes.

FAQs

Should engineers have SSH to production nodes at all? Increasingly no. kubectl debug node/<name> plus ephemeral containers cover nearly every case with better auditing.

Is crictl safe to run during an incident? The read commands are. Avoid rm, rmi, and stop, which conflict with the kubelet's reconciliation loop.

Why does ctr show nothing on my node? You omitted -n k8s.io. containerd namespaces are mandatory, and Kubernetes workloads live in that one.

Where should triage scripts live? In the platform repo, reviewed like any code, and baked into your debug image so the version on the node matches the version in git.

How do I debug a distroless image with no shell? Attach an ephemeral container with --target, or use nsenter from the host with only -n or -p so you keep host binaries.

What is the single highest-leverage practice here? Codify the checks. A read-only triage script in the repo outlives every individual's memory of what to look at.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Buscar en la documentación

CLI Best Practices