Service Mesh Best Practices
This is a practitioner checklist for deciding on, rolling out, and operating a service mesh on Kubernetes without letting it become the most fragile part of your platform.
The single most important practice sits at the top: do not adopt a mesh for a greenfield project before you have baseline Kubernetes maturity.
How to Use This List
Read the "Adopt only when justified" group first; if you fail it, stop and fix the prerequisites before touching a mesh.
Treat each bold item as a rule with a reason, and adapt it to whether you run Istio, Linkerd, or ambient mode.
Apply these incrementally, namespace by namespace, rather than mesh-wide on day one.
A - Adopt Only When Justified
- Meet baseline K8s maturity first. Have resource requests/limits, probes, NetworkPolicies, a metrics stack, and a tested upgrade process before adding a mesh; a mesh multiplies existing operational gaps.
- Do not mesh greenfield by default. A new project with a handful of services rarely needs one; start with Ingress or Gateway API plus NetworkPolicy and add a mesh when scale demands it.
- Tie adoption to real triggers. Adopt when you have many services, mixed languages, a fleet-wide mTLS mandate, or a need for L7 traffic shifting - not because it is fashionable.
- Compare against non-mesh options. For mTLS alone, weigh cert-manager or SPIRE; for metrics, weigh a shared OpenTelemetry standard. Pick the mesh only when uniformity at scale wins.
- Record the decision in an ADR. Capture context, decision, and consequences so the choice is reviewable and reversible.
B - Roll Out Incrementally
- Enroll one namespace at a time. Injection is opt-in; start with a low-risk namespace, validate, then expand.
- Keep system namespaces out. Do not mesh
kube-system, the mesh's own namespace, DNS, or the API path to avoid bootstrap deadlocks. - Restart workloads to inject them. Sidecar injection happens at admission, so annotating a namespace only affects pods after a rollout restart.
- Start mTLS permissive. Roll out in a permissive mode so plaintext callers keep working, then move to strict once every caller is meshed.
- Validate before widening. Use the mesh's check and proxy-status commands after each step before enrolling the next namespace.
C - Configure Security Deliberately
- Require strict mTLS once ready. After all callers are meshed, enforce strict mTLS so plaintext is rejected mesh-wide.
- Keep NetworkPolicy alongside the mesh. L3/L4 NetworkPolicy and L7 mesh authorization are complementary defense layers, not substitutes.
- Use default-deny authorization. Start L7 authorization from deny and allow specific service-account principals, not IP addresses.
- Do not conflate workload and user identity. mTLS authenticates workloads; keep application-level authn/authz for end users and API callers.
- Manage the trust anchor. Plan and rehearse root-certificate rotation; auto-rotated leaf certs do not cover the long-lived trust anchor.
D - Manage Performance and Cost
- Budget for per-pod overhead. Each sidecar adds memory, CPU, and a few milliseconds of latency per hop; multiply by pod count to estimate cost.
- Profile before and after injection. Measure real resource use with
kubectl toprather than assuming the overhead is negligible. - Consider ambient mode at scale. Sidecar-less L4 mesh lowers per-pod overhead and decouples proxy upgrades from app restarts.
- Set timeouts and retries conservatively. Cap retry attempts and only retry idempotent calls to avoid retry storms that amplify outages.
- Point metrics at durable storage. Bundled demo Prometheus is not durable; ship mesh metrics to your production observability stack.
E - Operate and Upgrade Safely
- Use revision-based upgrades. Run two control-plane revisions and migrate namespaces gradually rather than upgrading in place.
- Schedule fleet-wide restarts. Sidecar upgrades require re-injecting workloads; plan the rolling restart as a maintenance activity.
- Avoid single-person ownership. Ensure more than one engineer can operate and debug the mesh; skill concentration is an availability risk.
- Monitor the control plane. Alert on control-plane health and certificate expiry, since degradation stops new config and cert issuance.
- Keep an exit path. Document how to disable injection and remove strict-mTLS dependencies so the decision stays reversible.
When You Are Done
You should be able to name the concrete trigger that justified the mesh, and point to the ADR that records it.
Every meshed namespace should have strict mTLS, default-deny L7 authorization, and NetworkPolicies, with system namespaces excluded.
Your upgrade runbook should describe revision-based upgrades and the fleet restart they imply, and more than one engineer should be able to execute it.
If any of those are missing, treat the rollout as incomplete rather than done.
FAQs
What is the number-one mistake teams make?
Adopting a mesh on an immature platform. Without requests/limits, probes, NetworkPolicies, and a tested upgrade process, a mesh adds failure modes faster than benefits.
Should a greenfield project start with a mesh?
Usually not. Begin with Ingress or Gateway API plus NetworkPolicy, and introduce a mesh only when service count, language mix, or an mTLS mandate justifies it.
Do I still need NetworkPolicy if I run a mesh?
Yes. NetworkPolicy enforces L3/L4 isolation in the CNI; the mesh enforces L7 identity and policy. Keep both.
Is ambient mode always better than sidecars?
No. Ambient lowers per-pod overhead and eases upgrades, but it is newer and adds node-level components; choose based on maturity tolerance and needs.
How do I keep upgrades from causing outages?
Use revision-based canary upgrades, migrate namespaces one at a time, and schedule the required fleet restart during a maintenance window.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).