Runtimes Best Practices
This is a working checklist for teams operating Kubernetes at the runtime layer. It assumes containerd or CRI-O on nodes and Docker only in the build path.
How to Use This List
Treat each group as a review area for a node image, a cluster upgrade, or an incident retro.
Do not adopt every item blindly. Match them to your platform (managed vs self-run) and your threat model.
The highest-leverage habits are knowing your runtime and keeping build tooling off workers - start there.
A - Know and Standardize Your Runtime
Identify the runtime on every node. Run kubectl get nodes -o wide and confirm the CONTAINER-RUNTIME column; never assume Docker is involved.
Standardize one runtime per node pool. Mixing containerd and CRI-O across a pool complicates debugging and config; pick one and keep it uniform.
Pin the runtime version to the Kubernetes minor. CRI-O tracks Kubernetes one-to-one; containerd has a support matrix - stay inside it, especially across upgrades.
Document the runtime socket paths. containerd uses /run/containerd/containerd.sock, CRI-O uses /var/run/crio/crio.sock; put these in your runbooks so crictl just works.
Do not depend on dockershim. It was removed in Kubernetes 1.24 and does not exist in 1.36; any tooling assuming in-cluster Docker must be retired.
B - Keep Docker Off Your Workers
Never install Docker Engine on worker nodes. Nodes need a CRI runtime only; a second daemon adds attack surface, resource use, and confusion.
Build images in CI, not on nodes. Use BuildKit/buildx on runners; ship OCI images to a registry that nodes pull from.
Prefer daemonless build tools for in-cluster builds. If you must build inside the cluster, use BuildKit-in-pod, Kaniko, or Buildah rather than mounting the node's runtime socket.
Never mount the runtime socket into workloads. Exposing containerd.sock or the Docker socket to a pod is effectively root on the node.
Rely on the OCI handoff. Build with Docker, run with containerd - the shared OCI image format makes this seamless, so there is no reason to co-locate them.
C - Debug at the Right Layer
Reach for kubectl first. It is cluster-scoped and safe; only drop to the node when the control plane cannot answer.
Use crictl for CRI-level truth. When kubectl logs/exec fail, crictl ps, crictl logs, and crictl inspect talk straight to the runtime.
Remember the k8s.io namespace for ctr. containerd hides Kubernetes containers in the k8s.io namespace; pass --namespace k8s.io or you will see nothing.
Do not fight the kubelet with crictl rm. Deleting a runtime container the kubelet still wants just triggers recreation; delete the Kubernetes object instead.
Prefer kubectl debug node/... on managed clusters. It gives node access without SSH and without weakening node hardening.
D - Harden the Runtime Boundary
Run as non-root with dropped capabilities. Set runAsNonRoot, drop: ["ALL"], and allowPrivilegeEscalation: false so runc enforces a minimal process.
Enable seccomp by default. Apply seccompProfile: RuntimeDefault cluster-wide via Pod Security Standards (restricted) to shrink the syscall surface.
Ban privileged pods outside break-glass. privileged: true removes most isolation; gate it with policy and audit every use.
Use a sandboxed RuntimeClass for untrusted code. Route multi-tenant or untrusted workloads to gVisor or Kata via runtimeClassName, and schedule them onto nodes that have the handler.
Consider user namespaces. Mapping container root to an unprivileged host UID reduces the blast radius of an escape; enable it where supported.
E - Operate and Upgrade Safely
Pin images by digest. Use image@sha256:... in production so nodes pull immutable content and cannot drift.
Scan and sign images. Gate deploys on Trivy/Grype scans and verify Sigstore/cosign signatures before the runtime ever pulls them.
Test runtime upgrades on a canary pool. Roll containerd/CRI-O and node-image changes through a small pool before fleet-wide rollout.
Watch runtime health signals. Monitor kubelet and runtime metrics (image pull latency, container start errors) and alert on sandbox creation failures.
Manage runtime config through your platform. On OpenShift use ContainerRuntimeConfig; on managed clusters use node-pool settings - hand edits get reverted by reconcilers.
When You Are Done
You should be able to name the runtime and version on every node pool from memory or a dashboard.
No worker should have Docker installed, and no workload should mount a runtime socket.
Your restricted Pod Security Standard, seccomp default, and non-root policy should be enforced, with a sandboxed RuntimeClass ready for untrusted workloads.
FAQs
Do I need to choose containerd or CRI-O myself? Usually not - your distribution or cloud picks it. Managed clouds default to containerd; OpenShift uses CRI-O.
Is it ever OK to install Docker on a node? For a dedicated build node in CI, yes. For a Kubernetes worker running pods, no.
How do I enforce non-root and seccomp fleet-wide? Apply the restricted Pod Security Standard at the namespace level and reject pods that violate it in admission.
Which runtime is more secure? Neither containerd nor CRI-O is inherently more secure; both run runc/crun on a shared kernel. Real isolation gains come from gVisor or Kata.
What breaks when I upgrade the runtime? Rarely your images, since they are OCI. Watch for config format changes and version-matrix mismatches with the Kubernetes minor.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).