containerd & crictl

Summary

containerd is the high-level container runtime that the kubelet drives over CRI to pull images and run pods on a node.
crictl is a CRI-native CLI for inspecting and manipulating those pods and containers directly on the node.
Insight: When kubectl logs or kubectl exec fails, crictl often still works, because it talks to the runtime socket, not the API server.
Key Concepts: CRI socket, pod sandbox vs container, the k8s.io containerd namespace, crictl vs ctr.
When to Use: Node-level debugging, image-pull failures, stuck sandboxes, and confirming what the kubelet actually did.
Limitations: crictl is a debugging tool, not an orchestrator - it does not reconcile desired state and can leave the cluster inconsistent if misused.

Recipe

Get onto the node, point crictl at the runtime, and inspect.

# 1. Get a shell on the node (managed clusters: use a debug pod)
kubectl debug node/<node> -it --image=busybox
 
# 2. Point crictl at containerd
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
 
# 3. List sandboxes and containers
crictl pods
crictl ps -a

That is the whole loop: pods, containers, logs, inspect.

Working Example

Diagnose a pod stuck in ImagePullBackOff from the node.

# Confirm the pod sandbox exists but the container never started
crictl pods --name my-app
crictl ps -a --name my-app
 
# Try the pull manually to see the real error
crictl pull registry.example.com/my-app:1.4.2

If the pull fails, the error is now explicit - unauthorized, no such host, or manifest unknown.

Check the image credentials the node is using.

crictl inspecti registry.example.com/my-app:1.4.2 2>/dev/null || echo "not present on node"

For a running-but-crashing container, read logs and the full OCI config.

CID=$(crictl ps -a --name my-app -q | head -n1)
crictl logs "$CID"
crictl inspect "$CID" | jq '.status.exitCode, .info.runtimeSpec.process.args'

crictl inspect exposes the runtime spec: entrypoint args, mounts, env, seccomp, and the resolved user - the exact config the kubelet handed to containerd.

To rule out an image defect, run it in isolation with ctr.

ctr --namespace k8s.io images pull registry.example.com/my-app:1.4.2
ctr --namespace k8s.io run --rm registry.example.com/my-app:1.4.2 test /bin/true

If that succeeds, the image is fine and the problem is Kubernetes-level (secrets, RBAC, scheduling).

Deep Dive

containerd's architecture

containerd runs as a single daemon per node with a plugin architecture.

Its CRI plugin implements the gRPC interface the kubelet calls. Its content store holds image layers, deduplicated across pods.

Snapshotters (overlayfs by default) assemble the container root filesystem from those layers copy-on-write.

Each running container has a shim (containerd-shim-runc-v2) that owns the container's stdio and lifecycle, so containerd can restart without killing your workloads.

The k8s.io namespace

containerd has its own namespaces (unrelated to Linux namespaces) to isolate metadata.

Kubernetes puts everything under k8s.io. Always pass --namespace k8s.io to ctr, or you will see an empty list and think nothing is running.

crictl handles this for you, which is one reason to prefer it.

crictl vs ctr vs kubectl

kubectl is cluster-scoped and goes through the API server - your default tool.

crictl is node-scoped and CRI-level - use it when the control plane is unhealthy or you need the kubelet's ground truth.

ctr is containerd-internal and bypasses CRI - use it only to test containerd itself.

Configuring the endpoint

Set the endpoint once to avoid repetitive flags.

cat >/etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF

Without this, newer crictl prints a warning and probes default sockets, which is slower and noisier.

Gotchas

Empty ctr output. You forgot --namespace k8s.io. Kubernetes containers are invisible in the default namespace.

crictl rm fights the kubelet. If you delete a container the kubelet still wants, it recreates it immediately. Delete the Kubernetes object, not the runtime container.

Stopping a sandbox breaks pod networking. crictl stopp tears down the pod's network namespace. Only do it to intentionally reset a wedged pod.

Logs look truncated. crictl logs reads the current log file; rotated logs may be split. Use --tail and check the kubelet's log rotation settings.

Docker CLI is not installed. That is expected on containerd nodes. Reach for crictl/ctr, not docker.

Alternatives

nerdctl is a Docker-compatible CLI for containerd. It is friendlier than ctr (supports nerdctl ps, nerdctl run, Compose) and is great for local containerd or manual node work.

kubectl debug creates an ephemeral debug container or node debug pod without SSH - preferred on managed clusters where you cannot log into nodes.

Node problem detector / event tooling surfaces many runtime issues as Kubernetes events, so you often never need crictl at all.

Pick crictl when you need CRI ground truth, nerdctl for a Docker-like experience on containerd, and kubectl for everything routine.

FAQs

Is crictl installed by default? On many kubeadm and managed nodes yes, but not always. Install it from the cri-tools release matching your Kubernetes minor.

Can crictl start a pod? It can runp/create/start sandboxes and containers for testing, but it will not manage them like the kubelet. Treat it as read-mostly.

Why does docker ps fail on my worker? Because Docker is not the runtime and usually is not installed. containerd is, and crictl ps is the equivalent.

Does containerd store images separately from Docker? Yes. Docker Engine and containerd's CRI plugin use different content stores, even on a host that has both.

How do I match crictl to my cluster version? Use the cri-tools release with the same major.minor as Kubernetes (for example 1.36.x for a 1.36 cluster).

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

containerd & crictl

Summary

containerd is the high-level container runtime that the kubelet drives over CRI to pull images and run pods on a node.
crictl is a CRI-native CLI for inspecting and manipulating those pods and containers directly on the node.
Insight: When kubectl logs or kubectl exec fails, crictl often still works, because it talks to the runtime socket, not the API server.
Key Concepts: CRI socket, pod sandbox vs container, the k8s.io containerd namespace, crictl vs ctr.
When to Use: Node-level debugging, image-pull failures, stuck sandboxes, and confirming what the kubelet actually did.
Limitations: crictl is a debugging tool, not an orchestrator - it does not reconcile desired state and can leave the cluster inconsistent if misused.

Recipe

Get onto the node, point crictl at the runtime, and inspect.

# 1. Get a shell on the node (managed clusters: use a debug pod)
kubectl debug node/<node> -it --image=busybox
 
# 2. Point crictl at containerd
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
 
# 3. List sandboxes and containers
crictl pods
crictl ps -a

That is the whole loop: pods, containers, logs, inspect.

Working Example

Diagnose a pod stuck in ImagePullBackOff from the node.

# Confirm the pod sandbox exists but the container never started
crictl pods --name my-app
crictl ps -a --name my-app
 
# Try the pull manually to see the real error
crictl pull registry.example.com/my-app:1.4.2

If the pull fails, the error is now explicit - unauthorized, no such host, or manifest unknown.

Check the image credentials the node is using.

crictl inspecti registry.example.com/my-app:1.4.2 2>/dev/null || echo "not present on node"

For a running-but-crashing container, read logs and the full OCI config.

CID=$(crictl ps -a --name my-app -q | head -n1)
crictl logs "$CID"
crictl inspect "$CID" | jq '.status.exitCode, .info.runtimeSpec.process.args'

crictl inspect exposes the runtime spec: entrypoint args, mounts, env, seccomp, and the resolved user - the exact config the kubelet handed to containerd.

To rule out an image defect, run it in isolation with ctr.

ctr --namespace k8s.io images pull registry.example.com/my-app:1.4.2
ctr --namespace k8s.io run --rm registry.example.com/my-app:1.4.2 test /bin/true

If that succeeds, the image is fine and the problem is Kubernetes-level (secrets, RBAC, scheduling).

Deep Dive

containerd's architecture

containerd runs as a single daemon per node with a plugin architecture.

Its CRI plugin implements the gRPC interface the kubelet calls. Its content store holds image layers, deduplicated across pods.

Snapshotters (overlayfs by default) assemble the container root filesystem from those layers copy-on-write.

Each running container has a shim (containerd-shim-runc-v2) that owns the container's stdio and lifecycle, so containerd can restart without killing your workloads.

The k8s.io namespace

containerd has its own namespaces (unrelated to Linux namespaces) to isolate metadata.

Kubernetes puts everything under k8s.io. Always pass --namespace k8s.io to ctr, or you will see an empty list and think nothing is running.

crictl handles this for you, which is one reason to prefer it.

crictl vs ctr vs kubectl

kubectl is cluster-scoped and goes through the API server - your default tool.

crictl is node-scoped and CRI-level - use it when the control plane is unhealthy or you need the kubelet's ground truth.

ctr is containerd-internal and bypasses CRI - use it only to test containerd itself.

Configuring the endpoint

Set the endpoint once to avoid repetitive flags.

cat >/etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF

Without this, newer crictl prints a warning and probes default sockets, which is slower and noisier.

Gotchas

Empty ctr output. You forgot --namespace k8s.io. Kubernetes containers are invisible in the default namespace.

crictl rm fights the kubelet. If you delete a container the kubelet still wants, it recreates it immediately. Delete the Kubernetes object, not the runtime container.

Stopping a sandbox breaks pod networking. crictl stopp tears down the pod's network namespace. Only do it to intentionally reset a wedged pod.

Logs look truncated. crictl logs reads the current log file; rotated logs may be split. Use --tail and check the kubelet's log rotation settings.

Docker CLI is not installed. That is expected on containerd nodes. Reach for crictl/ctr, not docker.

Alternatives

nerdctl is a Docker-compatible CLI for containerd. It is friendlier than ctr (supports nerdctl ps, nerdctl run, Compose) and is great for local containerd or manual node work.

kubectl debug creates an ephemeral debug container or node debug pod without SSH - preferred on managed clusters where you cannot log into nodes.

Node problem detector / event tooling surfaces many runtime issues as Kubernetes events, so you often never need crictl at all.

Pick crictl when you need CRI ground truth, nerdctl for a Docker-like experience on containerd, and kubectl for everything routine.

FAQs

Is crictl installed by default? On many kubeadm and managed nodes yes, but not always. Install it from the cri-tools release matching your Kubernetes minor.

Can crictl start a pod? It can runp/create/start sandboxes and containers for testing, but it will not manage them like the kubelet. Treat it as read-mostly.

Why does docker ps fail on my worker? Because Docker is not the runtime and usually is not installed. containerd is, and crictl ps is the equivalent.

Does containerd store images separately from Docker? Yes. Docker Engine and containerd's CRI plugin use different content stores, even on a host that has both.

How do I match crictl to my cluster version? Use the cri-tools release with the same major.minor as Kubernetes (for example 1.36.x for a 1.36 cluster).

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

containerd & crictl

Search Documentation

containerd & crictl