Linux CLI Basics

This section covers the core Linux commands platform engineers use on Kubernetes worker nodes to debug the runtime and kernel below the pod.

Every example runs on the host, not inside a container, so you reach it via SSH, SSM, or kubectl debug node/<name>.

Prerequisites

• A Kubernetes 1.36.2 node with containerd as the CRI runtime.
• Host access (SSH, cloud session manager, or kubectl debug node/<name> for a privileged shell).
• crictl configured with the containerd socket; most distros ship it. If not:

# One-time crictl config so you can drop the --runtime-endpoint flag
cat >/etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF

• Root or sudo for nsenter, journalctl, and the cgroup filesystem.

Basic Examples

1. Find what is eating the node

One line to see CPU, memory, and load pressure on the host.

top -b -n1 | head -20

• -b runs in batch mode so output is greppable and scriptable.
• -n1 takes a single snapshot instead of refreshing.
• The load average line shows 1, 5, and 15 minute trends; compare against the node's vCPU count.
• Look for a single process pinning a core, which often means a runaway container.

2. Check disk pressure

Image pull failures and evictions frequently trace back to a full disk.

df -h /var/lib/containerd /var/lib/kubelet

• /var/lib/containerd holds image layers and container snapshots.
• /var/lib/kubelet holds emptyDir volumes and ephemeral storage.
• When either passes ~85%, the kubelet starts evicting pods for DiskPressure.
• Follow up with du -sh /var/lib/containerd/* | sort -h to find the culprit.

3. Inspect processes with their cgroup

Map a host PID back to the pod it belongs to.

ps -eo pid,ppid,cgroup,comm | grep -v '/system.slice' | head

• cgroup in the output reveals the kubepods slice, which encodes QoS class and pod UID.
• Guaranteed pods sit under kubepods.slice; Burstable and BestEffort get nested sub-slices.
• This links a misbehaving process to a specific workload without leaving the host.
• Pipe through grep <pod-uid> once you have the UID from kubectl.

4. Read node and runtime logs

The kubelet and containerd log to the systemd journal, not to files.

journalctl -u kubelet -u containerd --since "10 min ago" --no-pager

• -u selects a systemd unit; you can pass several to interleave them by timestamp.
• --since scopes the window so you are not scrolling boot history.
• Sandbox creation and image pull errors surface here first.
• Add -p err to show only error priority and above.

5. Watch kernel messages for OOM and driver events

The kernel ring buffer records OOM kills, network driver resets, and filesystem errors.

dmesg -T | grep -i -E 'oom|killed|blocked' | tail

• -T prints human-readable timestamps instead of seconds since boot.
• An oom-kill line names the killed process and its cgroup, confirming a memory limit hit.
• blocked for more than 120 seconds points to I/O stalls, often EBS or NFS latency.
• Correlate the timestamp with pod restart events from kubectl.

6. List containers at the runtime layer

crictl is the CRI-aware client that shows what containerd actually runs.

$ crictl ps -a
CONTAINER      IMAGE          STATE     NAME    ATTEMPT   POD
7f3a1b2c9d0e   nginx@sha256   Running   web     0         web-7d9f
2b8c4d1a6f5e   redis@sha256   Exited    cache   4         api-55c8

• -a includes exited containers, which reveals crash loops.
• The ATTEMPT column counts restarts at the runtime level.
• Use the container ID with crictl logs <id> and crictl inspect <id>.
• Unlike docker ps, this reflects the real pod runtime on a node.

7. Check open ports and sockets

Confirm whether a service is actually listening on the host or in a namespace.

ss -tlnp | head

• -t TCP, -l listening, -n numeric, -p shows the owning process.
• The kubelet listens on 10250; containerd exposes its socket, not a TCP port.
• A missing expected port often means the process crashed rather than a network issue.
• Run inside a pod's netns (see the nsenter page) to inspect per-pod listeners.

Intermediate Examples

8. Enter a running container's namespaces

Get a shell inside a container that has no shell of its own.

PID=$(crictl inspect --output go-template \
  --template '{{.info.pid}}' <container-id>)
nsenter -t "$PID" -n -p ip addr

• crictl inspect yields the host PID of the container's main process.
• nsenter -t <pid> -n enters just the network namespace to inspect its interfaces.
• Add -p -m to enter the PID and mount namespaces for a fuller view.
• This works even when kubectl exec fails because the kubelet is unhealthy.

9. Read cgroup v2 memory accounting

See real-time memory usage and OOM events for a pod, straight from the kernel.

cd /sys/fs/cgroup/kubepods.slice/<pod-slice>
cat memory.current memory.max memory.events

• memory.current is live usage in bytes; memory.max is the enforced limit.
• memory.events counts oom and oom_kill occurrences since the cgroup was created.
• A rising oom_kill count proves the workload needs a higher limit or a leak fix.
• cgroup v2 is the unified hierarchy default on Kubernetes 1.36 nodes.

10. Trace what a stuck process is doing

When a container hangs, see which syscall it is blocked on.

strace -f -p "$PID" -e trace=network,file 2>&1 | head

• -f follows child threads; -p attaches to the running PID from step 8.
• -e trace=network,file filters to the syscalls that matter for hangs.
• A process stuck in connect() or futex() tells you it is a network or lock issue.
• Detach with Ctrl-C; strace adds overhead, so keep the attach brief.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).