Shift-Left vs Runtime Security
Summary
- Definition: Shift-left security moves checks earlier - into the Dockerfile, CI pipeline, and admission gate - while runtime security watches containers as they execute in the cluster.
- Insight: These are not two competing tools you choose between. They cover different time windows and different threat classes, so mature platforms run both.
- Key Concepts: image scanning, software bill of materials (SBOM), admission control, runtime detection, behavioral drift, defense in depth.
- When to Use: Shift-left for known vulnerabilities and misconfiguration before deploy; runtime for exploitation, privilege escalation, and unexpected process or network activity after deploy.
- Limitations/Trade-offs: Shift-left cannot see zero-days or attacker behavior; runtime cannot un-ship a vulnerable image and adds observation overhead on every node.
- Related Topics: Trivy/Grype scanning, admission gating, Falco, eBPF observability.
Foundations
Container security spans a timeline: an image is built, scanned, admitted, then run.
Shift-left means catching problems as far left on that timeline as possible - ideally in a pull request, before anything reaches a node.
The economics are simple. A vulnerability caught in CI costs a rebuild; the same vulnerability exploited in production costs an incident response.
Runtime security lives at the far right of the timeline. It assumes something got through and watches for the consequences.
The mental model is a building with locks and cameras. Locks (shift-left) stop known-bad entry. Cameras (runtime) tell you when someone got in anyway.
Neither replaces the other. A locked building with no cameras is blind to insiders and lock-picking; cameras with no locks just record every break-in.
Mechanics & Interactions
Shift-left has three practical stages.
First, image scanning. Tools like Trivy and Grype read the image layers, enumerate installed OS packages and language dependencies, and match them against vulnerability databases keyed to versions.
Second, SBOM generation. A scanner produces a machine-readable inventory (SPDX or CycloneDX) so you can later answer "which running images contain this library" without re-scanning everything.
Third, admission control. A Kubernetes admission webhook - or a policy engine like Kyverno or an OPA Gatekeeper constraint - inspects each Pod spec at deploy time and can deny images with critical CVEs, unsigned images, or unsafe securityContext settings.
# Kyverno: block Pods whose images are not signed by our cosign key
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-images
spec:
validationFailureAction: Enforce
rules:
- name: verify-signature
match:
any:
- resources:
kinds: ["Pod"]
verifyImages:
- imageReferences: ["registry.example.com/*"]
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----Runtime security works differently. It does not read manifests; it observes syscalls and kernel events on the node.
Falco, the CNCF runtime detection engine, consumes a kernel event stream (via an eBPF probe) and evaluates rules. A rule might fire when a shell is spawned inside a container that should never run one.
eBPF is the shared enabler. It lets tools attach safe, sandboxed programs to kernel hooks and see process, file, and network events without patching the kernel or adding a sidecar per Pod.
Cilium Hubble builds on the same foundation for network visibility, surfacing which service talked to which and whether a NetworkPolicy allowed or dropped the flow.
The two halves interact through feedback. Runtime detections should feed back into shift-left: a Falco alert about an exploited package tells you which scan rule to tighten and which image to rebuild.
Advanced Considerations & Applications
Coverage gaps are the reason both layers exist.
Shift-left is blind to anything not yet in a vulnerability database. A zero-day, a supply-chain backdoor in a trusted dependency, or a logic flaw in your own code all pass scanning cleanly.
Runtime is blind to intent before action. It cannot tell you an image is vulnerable; it can only tell you when that vulnerability is being exploited, by which time you are responding rather than preventing.
Behavioral drift is the concept that ties runtime detection to real risk. A container's legitimate behavior is narrow - a web server binds a port, reads config, serves requests. Anything outside that envelope (spawning bash, writing to /etc, opening an outbound connection to a new IP) is drift worth alerting on.
In production, tune for signal. Falco out of the box is noisy; you scope rules to namespaces, exclude known-good exec patterns (a debugging sidecar, a legitimate init job), and route alerts to a system that can page.
Cost matters too. Runtime observation runs on every node and consumes CPU proportional to event volume; a chatty workload with high syscall rates costs more to watch than an idle one.
A defense-in-depth posture layers all of it: non-root pinned base images, CI scanning that fails the build on criticals, an admission gate that denies unsigned or unscanned images, restricted Pod Security Standards, default-deny NetworkPolicies, and Falco plus Hubble watching what actually runs.
Common Misconceptions
"If we scan images, we don't need runtime detection." Scanning only knows about published CVEs. Exploitation of a zero-day or misuse of valid credentials leaves scanning untouched but is exactly what runtime detection sees.
"Runtime security means we can skip scanning." Detecting an exploit in production is far more expensive than rejecting the vulnerable image in CI. Runtime is a backstop, not a substitute for prevention.
"Admission control is runtime security." Admission runs once, at deploy time, on the Pod spec. It is a shift-left gate. Once the Pod is running, admission has no further visibility.
"eBPF tools need a sidecar in every Pod." eBPF programs attach at the kernel/node level and observe all containers on that node from one agent per node, typically a DaemonSet - no per-Pod sidecar required.
"A green scan means the image is safe." A clean scan means no known vulnerabilities in scanned components. Your own application code, secrets baked into layers, and undisclosed flaws are all outside its scope.
FAQs
Is shift-left just scanning? No. It includes Dockerfile hardening, SBOM generation, image signing, and admission policies - scanning is one stage of a broader pre-deploy discipline.
Where does image signing fit? It is shift-left. Signing (cosign) plus admission verification proves an image came from your pipeline, defending against registry tampering and typosquatting.
Can one tool do both? Some platforms bundle scanning and runtime, but they are distinct engines internally - a scanner reading layers versus a kernel-event detector. Treat them as separate capabilities.
Does runtime security need privileged access? The node agent needs elevated privileges to load eBPF programs, but this is one hardened DaemonSet, not privilege granted to your workloads.
What should fire an admission deny? Critical/High CVEs above your threshold, unsigned images, missing SBOM attestation, and Pod specs that violate restricted Pod Security Standards.
How do the two layers share data? Through SBOMs and alerts. An SBOM lets you map a new CVE to running images; a runtime alert tells you which scan rule failed to prevent an exploit.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).