crictl & ctr
Summary
crictlis a CLI that speaks the Container Runtime Interface (CRI) to containerd, letting you list and inspect the pods and containers Kubernetes actually created on a node.- Insight:
docker psis not available on a modern node. dockershim was removed in Kubernetes 1.24, so containerd runs your pods andcrictlis the equivalent tool. - Key Concepts: CRI (the kubelet-to-runtime API), pod sandbox (the pause container holding the network namespace), namespace (containerd's tenancy boundary,
k8s.iofor CRI workloads), snapshotter (how layers become a rootfs). - When to Use: when the API server has no answer -
ImagePullBackOffwith no event detail, a container that restarts before logs flush, a pod stuck inContainerCreating. - Limitations:
crictlis a debugging tool, not an orchestrator. Containers you create with it are invisible to the kubelet and may be garbage collected.
Recipe
Point crictl at the containerd socket once, then list and inspect.
# Write /etc/crictl.yaml so every later command is short
cat >/etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF
crictl pods # pod sandboxes on this node
crictl ps -a # containers, including exited ones
crictl images # images in the CRI image storeWithout the config file, pass --runtime-endpoint unix:///run/containerd/containerd.sock on every invocation.
Working Example
A pod is CrashLoopBackOff and kubectl logs returns nothing useful because the process dies instantly.
Drop to the node and find the sandbox by name.
$ crictl pods --name checkout-api --namespace payments
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
9f2c1d4a7b3e 4 minutes ago Ready checkout-api-7d9f6-x2kql payments 0 (default)List the containers inside that sandbox, including dead ones.
$ crictl ps -a --pod 9f2c1d4a7b3e
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
b71e0c8f2a19 sha256:4c9b… 2 minutes ago Exited api 6 9f2c1d4a7b3eRead the logs of the exited attempt, which the kubelet may have already rotated away.
crictl logs --previous b71e0c8f2a19
crictl logs --tail 50 -f b71e0c8f2a19Inspect the container to see the exit code, the resolved command, and the mounts.
crictl inspect b71e0c8f2a19 | jq '.status.exitCode, .status.reason, .info.config.command'An exit code of 128+n means a signal killed it; 137 is SIGKILL, which usually means the OOM killer or a failed liveness probe.
If the image is the suspect, verify it exists on the node and pull it manually.
crictl images | grep checkout
crictl pull ghcr.io/acme/checkout-api@sha256:8f1c...A manual crictl pull reproduces registry auth and network failures with a real error message instead of a terse pod event.
Deep Dive
crictl vs ctr vs nerdctl
crictl talks the CRI API, so it sees exactly what the kubelet sees - sandboxes, containers, and the CRI image store.
ctr is containerd's own low-level client, shipped with containerd itself. It bypasses CRI entirely and speaks to containerd's native API.
nerdctl is the Docker-compatible CLI for containerd. It is convenient but not installed on most managed node images, so do not depend on it.
Reach for crictl first. Drop to ctr only when you need something below CRI, like raw snapshot or namespace inspection.
containerd namespaces
ctr requires an explicit namespace because containerd multiplexes tenants. Kubernetes workloads live in k8s.io.
$ ctr namespaces list
NAME LABELS
k8s.io
moby
$ ctr -n k8s.io containers list | head -5Forgetting -n k8s.io is the single most common ctr mistake - you get an empty list and conclude, wrongly, that the node is idle.
Inspecting the image store with ctr
ctr shows layer and content detail that CRI does not surface.
ctr -n k8s.io images list | grep checkout
ctr -n k8s.io content ls | head
ctr -n k8s.io snapshots usagesnapshots usage is how you attribute disk consumption when a node hits DiskPressure and image GC is not keeping up.
Pod sandboxes and the pause container
Each pod has a sandbox: the pause container that holds the network and IPC namespaces the app containers join.
crictl pods lists sandboxes; crictl ps lists the app containers within them. A sandbox in NotReady with no containers means CNI failed to wire the network.
crictl inspectp 9f2c1d4a7b3e | jq '.status.network'That returns the pod IP the runtime assigned, which is the ground truth when the API server reports something different.
Exec and stats
crictl exec works when kubectl exec cannot reach the API server or the pod lacks a shell path the API expects.
crictl exec -it b71e0c8f2a19 sh
crictl stats --id b71e0c8f2a19crictl stats reads live cgroup counters, so it reflects the kernel's view rather than a scraped metric that may be a minute stale.
Gotchas
Creating containers with ctr run on a live node. The kubelet does not know about them, they escape resource accounting, and they can linger after a rollout. Use them only in a throwaway debug context.
Assuming crictl rmi frees disk permanently. The kubelet will re-pull on the next scheduling decision. Fix the image GC thresholds (imageGCHighThresholdPercent) instead.
Empty ctr output. You almost certainly omitted -n k8s.io.
crictl logs after the kubelet rotated the file. CRI logs are files under /var/log/pods; once rotated out, only your log backend has them. Ship logs off-node.
Version skew warnings. crictl prints a warning if its version is far from the runtime's. It is usually harmless, but pin crictl near your Kubernetes minor.
Alternatives
kubectl debug node/<name> gives you a privileged pod on the node with host mounts, no SSH required. Prefer it when SSH is locked down, and it is the safest default for most teams.
nerdctl offers Docker-like ergonomics over containerd. Good on a dev box, unreliable to assume on managed nodes.
journalctl -u containerd answers runtime-level questions - CNI plugin errors, snapshotter failures - that no per-container tool exposes.
kubectl get events and pod status remain the first stop. Only drop to the node when the control plane's answer is empty or contradicts reality.
FAQs
Can I still use docker ps on a Kubernetes node?
No. Since Kubernetes 1.24 removed dockershim, containerd runs pods directly and Docker Engine is not installed on the node.
Does Docker Engine 29.6.1 still matter then? Yes, for building images and local Compose workflows. It is a build and dev tool, not the in-cluster runtime.
Why does crictl ps show fewer containers than I expect?
crictl ps hides exited containers by default. Add -a.
Is crictl safe to run on a production node?
Read commands like pods, ps, inspect, logs, and stats are safe. Mutating commands such as rm, rmi, and stop fight the kubelet and should be avoided.
How do I map a crictl container back to a Kubernetes pod?
crictl inspect <id> | jq '.info.config.labels' returns the io.kubernetes.pod.name and io.kubernetes.pod.namespace labels.
Where do CRI container logs live on disk?
Under /var/log/pods/<namespace>_<pod>_<uid>/<container>/, with symlinks from /var/log/containers.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).