Why Rules Beat Tribal Knowledge

Summary

• Definition: Container rules are the codified, versioned standards that turn one-off Docker and Kubernetes lessons into repeatable, enforceable defaults for every team.
• Insight: Knowledge that lives only in a senior engineer's head is a single point of failure - it does not scale, it does not survive attrition, and it cannot gate a pull request.
• Key Concepts: tribal knowledge, shift-left enforcement, policy as code, admission control, golden paths, versioned standards.
• When to Use: The moment a second team ships containers, or the first time a preventable incident recurs because a lesson was never written down.
• Limitations/Trade-offs: Rules add friction and can ossify; they must be maintained, exception-tracked, and periodically pruned or they become cargo cult.
• Related Topics: Dockerfile rules, Kubernetes manifest rules, policy engines, and architecture decision records.

Foundations

Tribal knowledge is the unwritten set of "how we do things here" that lives in Slack threads, code review comments, and the memory of whoever was on call last quarter.

It feels efficient because experienced people already know it. It is actually fragile because new hires, contractors, and adjacent teams do not.

A rule is different. It is an explicit, agreed statement of intent - "images run as non-root," "every workload sets requests and limits" - that has a name, an owner, and a version.

The mental model is a ratchet. Each time you learn something painful, you turn the ratchet one notch and encode the lesson so the whole organization can never slip back below that line.

Docker and Kubernetes are unusually good candidates for this ratchet. Their failure modes are well-known, repeatable, and mostly preventable with a handful of defaults.

Running as root, unpinned base tags, missing resource limits, absent liveness and readiness probes, and secrets baked into image layers cause the same incidents at company after company.

The tribal approach relearns each lesson per team, per person, forever. The rules approach learns it once and enforces it automatically.

Mechanics & Interactions

Rules become valuable only when they move from a wiki page to a place that can block bad changes. That progression is usually described as shift-left enforcement.

The leftmost, cheapest layer is the developer's machine. A pre-commit hook or a linter like hadolint for Dockerfiles catches an unpinned base or a root user before code is even pushed.

# Fail fast in local dev and CI
hadolint Dockerfile
kubeconform -strict -summary manifests/

The next layer is CI. Here you scan images with Trivy or Grype, validate manifests with kubeconform, and run policy checks with Conftest or kyverno in test mode.

# Block the merge if a HIGH/CRITICAL CVE is present
trivy image --exit-code 1 --severity HIGH,CRITICAL registry.example.com/api:1.4.2

The rightmost, last-resort layer is the cluster itself. A Kubernetes admission controller - Kyverno or Gatekeeper (OPA) - rejects any Pod that violates policy at the API server, regardless of how it was submitted.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-nonroot
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-nonroot
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "Containers must set runAsNonRoot: true"
        pattern:
          spec:
            containers:
              - securityContext:
                  runAsNonRoot: true

These layers interact deliberately. The left layers give fast, friendly feedback; the right layer is the non-bypassable backstop that makes the rule true even for the pipeline you forgot about.

Pod Security Standards fit the same pattern. Labeling a namespace pod-security.kubernetes.io/enforce: restricted turns a written rule into a cluster-enforced one without any custom policy code.

Advanced Considerations & Applications

The goal is not maximum enforcement; it is the right enforcement at the right cost. Every rule you enforce should trace back to a real failure mode or compliance requirement.

Rules pair naturally with golden paths - the paved, supported way to build and ship a service. When the default template already passes every rule, compliance becomes the path of least resistance rather than a tax.

Exceptions are a first-class feature, not a failure. A mature system lets a team request a scoped, time-boxed exception with an owner and an expiry, tracked in the same repository as the rules.

Kyverno and Gatekeeper both support this through policy exclusions and audit mode, letting you roll a new rule out in Audit before flipping it to Enforce.

# Stage a new rule safely: report violations without blocking
spec:
  validationFailureAction: Audit

Versioning matters because clusters and images move. A rule written for the dockershim era is wrong today - Docker Engine is not the in-cluster runtime; containerd runs pods via the CRI on nodes.

Treat the rule set like any other product. It has a changelog, a review cadence, and deprecations, and it is pinned to specific stack versions so "correct" is unambiguous.

Finally, rules are a leadership tool. They encode judgment, so senior engineers stop repeating themselves in review and spend their attention on genuinely novel decisions.

Common Misconceptions

"Rules slow us down." Unenforced ambiguity slows you down more - through review churn, inconsistent services, and incidents. Good rules with golden paths make the safe thing the fast thing.

"Our seniors already know this, so we do not need rules." Seniors leave, get busy, or go on vacation. A rule survives all three; a mental note does not.

"Docker runs my containers in the cluster." Not since dockershim was removed. You build with Docker and BuildKit, but containerd executes pods on nodes through the CRI.

"A wiki page is a rule." A wiki page is documentation. A rule is something that can block a merge or an admission request; if nothing enforces it, it is a suggestion.

"More policies mean more safety." Beyond a point, unused or redundant policies create noise, false positives, and exception fatigue. Prune aggressively.

FAQs

Where should a rule be enforced first? As far left as is practical - linters and CI - with a cluster-side admission backstop for the rules that must never be bypassed.

Do I need Kyverno or Gatekeeper if I use Pod Security Standards? PSS covers common pod hardening. Use a policy engine for anything PSS does not express, such as registry allow-lists or required labels.

How do we avoid rules becoming stale? Version them, pin them to stack versions, review them on a cadence, and delete rules whose failure mode no longer exists.

What about legitimate exceptions? Make them explicit, scoped, owned, and time-boxed, tracked in the rules repo and expressed as audit-mode or policy exclusions rather than silent overrides.

How many rules should we start with? Start with the short list that prevents your most common incidents - non-root, pinned bases, resource limits, probes, no secrets in layers - and grow from evidence.

Who owns the rules? A platform or enablement team usually stewards them, but the rules should be authored and reviewed with the application teams that live under them.

Related

• Containers Rules Checklist
• Dockerfile Rules
• Kubernetes Manifest Rules
• ADR Template
• Containers Rules Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).