Services Best Practices
How to Use This List
These are the habits that keep Kubernetes Services reliable, cheap, and debuggable at scale.
Treat each group as a checklist you can apply during design review and before a rollout.
They assume the pinned stack and the default kube-proxy or CNI dataplane.
Adopt the ones that fit your workload; none of them are exotic.
A - Naming and Ports
Always name your ports. A name: http on the Service and container lets probes, other Services, and EndpointSlices reference the port symbolically instead of by number.
Target ports by name, not number. Use targetPort: http so renumbering a container port does not require editing every Service.
Split traffic and metrics ports. Keep an http port for real traffic and a separate metrics port for Prometheus, so scrape traffic and app traffic are independent.
Set an explicit protocol only when needed. TCP is the default; declare UDP or SCTP deliberately and never by accident.
B - Health, Readiness, and Draining
Define a readiness probe on every Service-backed pod. Endpoints follow readiness, so an unready pod is pulled from EndpointSlices without being killed.
Point readiness at a cheap, dependency-free path. A probe that calls a database will flap the whole Service when that dependency hiccups.
Document gRPC vs HTTP probe differences. HTTP services use httpGet; gRPC services should use the native grpc probe or a gRPC health endpoint, not a bare TCP check that reports healthy on a broken app.
Add a preStop hook and graceful termination. Let in-flight requests finish while the endpoint drains, using the serving/terminating conditions the platform already tracks.
C - Load Balancing and Protocols
Know that L4 Services balance per-connection. A single keep-alive, HTTP/2, or gRPC connection pins to one pod, so raw ClusterIP will not spread multiplexed traffic.
Use client-side load balancing or a mesh for gRPC. This spreads requests across backends instead of concentrating them on one replica.
Reach for Ingress or Gateway API for L7 routing. Per-request routing on host, path, or method belongs at L7, not on a bare Service.
Set sessionAffinity: ClientIP only when you need stickiness. It keys routing on source IP for a timeout; otherwise leave it off for even spreading.
D - External Exposure and Cost
Preserve client IP with externalTrafficPolicy: Local. The default Cluster policy SNATs cross-node traffic and hides the caller IP.
Consolidate LoadBalancers behind Ingress or Gateway API. One cloud LB per Service is costly; share a single external LB across many hostnames.
Firewall NodePorts. NodePort opens a high port on every node IP, so restrict it with security groups or NetworkPolicy.
Cut cross-zone traffic with trafficDistribution: PreferClose. Prefer same-zone endpoints to reduce inter-zone data charges while keeping cross-zone fallback.
E - Security and Policy
Default-deny with NetworkPolicy, then allow explicitly. A Service does not restrict who can reach a pod; policy does.
Scope Service reach by namespace and DNS. Rely on <service>.<namespace> names so cross-namespace access is intentional.
Do not expose internal Services externally by habit. Keep internal APIs on ClusterIP and route external traffic through a controlled edge.
Align LB health checks with readiness probes. Divergent checks route traffic to pods your cluster considers unhealthy.
F - Observability and Debugging
Check EndpointSlices first when traffic disappears. An empty endpoint set means the selector matches nothing or pods are not Ready.
Label Services and pods consistently. Selectors and slice queries (kubernetes.io/service-name=) depend on clean, predictable labels.
Watch slice churn during large scale events. Fast autoscaling produces many slice updates that can stress the control plane and kube-proxy.
Verify DNS as a distinct step. Resolve the Service name explicitly so you can separate DNS failures from routing failures.
Distinguish ready from serving when draining. An endpoint with serving: true, ready: false during termination is intentional, not a defect - it lets in-flight requests finish.
Prefer per-endpoint node names for locality debugging. EndpointSlices carry the nodeName, which helps explain uneven load or zone-skewed traffic.
When You Are Done
You should be able to point at any Service and name its ports, its readiness contract, and how it is exposed.
External Services should share load balancers and preserve client IP where it matters.
EndpointSlices, NetworkPolicy, and DNS should all be part of your debugging muscle memory.
If any of those is unclear, revisit the group above that covers it.
FAQs
Why name ports if numbers work? Names decouple the Service from container port numbers and make probes, metrics, and slices readable.
Why does gRPC need special probe and LB handling? gRPC multiplexes many requests over one HTTP/2 connection, which L4 Services pin to a single pod, and a TCP probe cannot tell if the gRPC app is actually healthy.
How do I keep the real client IP?
Set externalTrafficPolicy: Local on NodePort or LoadBalancer Services to avoid cross-node SNAT.
What is the cheapest way to expose many services? Put an Ingress controller or Gateway API in front so they share one cloud load balancer.
Where do I look when a Service returns no traffic? Start with EndpointSlices and readiness, then verify DNS resolution and NetworkPolicy.
Should every service be a LoadBalancer? No. Keep internal services on ClusterIP and route external traffic through a shared Ingress or Gateway to control cost and exposure.
When is session affinity worth the trade-off? Only when a workload genuinely needs a client pinned to one backend; otherwise it undermines even load spreading.
Related
- Why Pods Need Services
- Services Basics
- Life of a Request
- ClusterIP & Headless
- NodePort & LoadBalancer
- EndpointSlices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).