Tech Lead Basics on Platforms

This page is the practical starter kit for a tech lead who now owns a shared Kubernetes platform. It shows the concrete artifacts - templates, defaults, and gates - that let teams ship fast without putting the cluster at risk.

Prerequisites

• Kubernetes 1.36.2 cluster access with a namespace you administer.
• kubectl matching the cluster minor, plus Helm 3 and Kustomize (bundled in kubectl).
• Docker Engine 29.6.1 with BuildKit and Compose v2 for local build/dev.
• A GitOps controller (Argo CD or Flux) if you want deploys from Git, not laptops.

Quick sanity check before you start:

kubectl version
kubectl auth can-i create deployments -n team-payments
helm version

Basic Examples

1. Set a safe default with resource requests and limits

Every workload gets requests and limits so one service cannot starve a node.

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"

• Requests drive scheduling; limits cap runaway usage.
• Set memory limit equal to request to avoid OOM surprises under pressure.
• Missing requests is the top cause of noisy-neighbor incidents.
• Bake this into your template so nobody has to remember it.

2. Add liveness and readiness probes

Probes let Kubernetes route traffic and restart hung pods correctly.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10

• Readiness gates traffic; a failing pod is pulled from the Service.
• Liveness restarts a wedged process, but a bad liveness probe causes restart loops.
• Keep the checks cheap - do not query the database in a liveness probe.
• Without readiness, deploys send traffic to pods that are not ready.

3. Run as non-root with a restricted security context

The paved road ships pods that cannot escalate privileges.

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: ["ALL"]

• Matches the Pod Security Standards restricted profile.
• Blocks most container-breakout techniques by default.
• Build images with a non-root USER so this does not fail at start.
• Enforce it cluster-wide later; ship it in the template now.

4. Pin base images by digest

Reproducible builds start with a base you can name exactly.

# pin by digest, not a moving tag
FROM node:22-slim@sha256:0a1b2c3d4e5f...
USER node

• Tags like latest move under you and break reproducibility.
• Digests make the build auditable and cacheable.
• Scan the pinned image with Trivy or Grype in CI.
• Update digests deliberately, through a PR, not silently.

5. Label workloads so ownership is discoverable

Standard labels make triage and cost attribution possible.

metadata:
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/part-of: payments
    team: payments

• app.kubernetes.io/* are the recommended common labels.
• A team label routes alerts and splits cost reports.
• Enforce required labels in review or admission policy.
• Unlabeled resources become orphans nobody will claim at 2am.

6. Deploy through Git, not kubectl apply from a laptop

GitOps gives you an audit trail and easy rollback.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
spec:
  destination:
    namespace: team-payments
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/acme/payments-manifests
    path: overlays/prod
    targetRevision: main
  syncPolicy:
    automated:
      selfHeal: true

• Git is the source of truth; the cluster reconciles to match.
• selfHeal reverts manual drift automatically.
• Rollback is a git revert, not a frantic manual edit.
• No human needs cluster write access for routine deploys.

7. Give each team a namespace with a quota

Quotas cap blast radius and make capacity predictable.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    pods: "50"

• One team cannot consume the whole cluster by accident.
• Pair with a LimitRange to set per-pod defaults.
• Quotas force conversations about capacity before an incident.
• Namespaces are the cheapest isolation boundary you have.

Intermediate Examples

8. Enforce the paved road with an admission policy

Guardrails reject unsafe specs no matter who wrote them.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-resource-requests
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "CPU and memory requests are required."
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"

• Policy runs at admission, before the pod is scheduled.
• Enforce blocks; use Audit first to measure impact.
• This backstops the template for people who bypass it.
• Start with one or two rules, not fifty.

9. Default-deny network traffic, then allow explicitly

A closed default limits lateral movement after a compromise.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

• With nothing allowed, every needed path becomes explicit.
• Add allow rules per dependency, documented in review.
• Requires a CNI that enforces NetworkPolicy (Cilium, Calico).
• Default-open networking is a standing incident waiting to happen.

10. Keep local build close to cluster runtime

Match your Compose dev loop to what containerd runs in prod.

docker build -t payments-api:dev .
docker compose up
# pods on nodes are started by containerd via the CRI,
# so keep the same image and same non-root user everywhere

• Build with Docker/BuildKit; run pods with containerd on nodes.
• Same image digest local and prod kills "works on my machine".
• Test the restricted security context locally before deploy.
• Do not assume Docker Engine runs your pods - it does not.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).