Life of a Request
Summary
- This page traces a packet from a client, through the Service VIP and kube-proxy, to a real endpoint pod.
- Insight: In default mode there is no proxy process in the data path - the kernel rewrites the destination and the packet flows straight to a pod.
- Key Concepts: DNS resolution, ClusterIP, DNAT, conntrack, kube-proxy modes.
- When to Use: Whenever you are debugging why traffic lands on the wrong pod, drops, or spreads unevenly.
- Limitations/Trade-offs:
iptablesmode balances per-connection; large Services stress rule count, whichIPVSand eBPF address. - Related Topics: Why pods need Services, EndpointSlices, NodePort and LoadBalancer.
Foundations
Start with the simplest case: one pod calling a ClusterIP Service in the same cluster.
The client code opens a connection to a name, say http://payments.
Before any packet leaves, the name must become an IP.
The application asks the resolver, which is CoreDNS, and gets back the Service's ClusterIP.
That ClusterIP is virtual - no NIC owns it.
The client now sends a normal TCP SYN to the ClusterIP.
The magic happens as that packet traverses the sending node's kernel.
kube-proxy (or a CNI dataplane) has already programmed rules that match the ClusterIP and rewrite it to a real pod IP.
From there the packet is routed pod-to-pod by the CNI, exactly like any other cluster traffic.
The mental model: DNS picks the door, the kernel picks the room, the CNI walks the packet there.
Mechanics & Interactions
Break the path into stages.
Stage 1 - DNS. CoreDNS resolves payments.default.svc.cluster.local to the ClusterIP. This is cached per the record TTL.
Stage 2 - endpoint selection. kube-proxy watches EndpointSlices for the Service and knows the current Ready pod IPs.
Stage 3 - DNAT. When the SYN hits the ClusterIP, a kernel rule performs Destination NAT, rewriting the destination to a chosen pod IP and port.
In iptables mode the choice uses statistical match rules that approximate even spreading across endpoints.
Stage 4 - conntrack. The kernel's connection tracking table records the mapping, so every later packet in that flow goes to the same pod and the reply is un-NATed correctly.
Stage 5 - delivery. The CNI routes the rewritten packet to the destination pod, which sees a normal connection.
The reply travels back, conntrack reverses the translation, and the client sees the ClusterIP as the source, preserving the illusion.
The whole exchange is per-connection: once conntrack pins a flow, it stays on that pod.
Advanced Considerations & Applications
kube-proxy runs in one of a few modes, and the mode changes the dataplane, not the semantics.
iptables mode is the long-standing default and is simple and reliable, but rule evaluation and updates scale roughly with the number of Services and endpoints.
IPVS mode uses the kernel's in-kernel load balancer with hash tables, giving faster lookups and real scheduling algorithms (round-robin, least-connection) for very large clusters.
eBPF dataplanes such as Cilium can replace kube-proxy entirely, doing the DNAT in eBPF programs and often skipping conntrack overhead.
Semantics you can rely on across modes: connection affinity, readiness-driven endpoint membership, and sessionAffinity: ClientIP if you set it.
External traffic adds a hop.
A LoadBalancer sends packets to a node's NodePort, and with externalTrafficPolicy: Cluster the node may DNAT to a pod on another node, doing SNAT so replies return - which hides the client IP.
externalTrafficPolicy: Local keeps traffic on the receiving node and preserves the client IP, at the cost of uneven distribution.
For per-request routing (HTTP paths, gRPC methods) you step up to Ingress, Gateway API, or a mesh, which terminate L7 and balance each request.
DNS behavior is worth tracing too, because it is a common failure point.
The client's /etc/resolv.conf points at the CoreDNS ClusterIP and includes search domains like default.svc.cluster.local.
A short name such as payments is expanded through those search domains until one resolves.
That is why a name that works in one namespace can silently resolve to a different Service in another.
Record TTLs also mean a client may keep an old resolution briefly after a Service changes, so DNS is not instantaneous.
Common Misconceptions
"Traffic flows through the kube-proxy process." It does not. kube-proxy programs kernel rules and then steps aside; the data path is entirely in-kernel.
"Each request is load balanced independently." In L4 modes, balancing is per-connection. A reused connection stays pinned, which is why keep-alive and gRPC concentrate on one pod.
"DNS returns pod IPs." For a normal Service, DNS returns the ClusterIP. Only headless Services return pod IPs.
"The client IP is always visible to the pod." With the default Cluster external policy, SNAT can replace the client IP. Use Local to preserve it.
"kube-proxy routes pod-to-pod traffic." No. The CNI handles pod networking. kube-proxy only implements the Service VIP translation.
FAQs
Why does one pod get most of my traffic? Long-lived or multiplexed connections (keep-alive, HTTP/2, gRPC) pin to one backend. Spread load with client-side balancing or a mesh.
Where is the ClusterIP actually configured? Nowhere as an interface. It exists only as forwarding rules that kube-proxy or the CNI installs on every node.
How do I make a client always reach the same pod?
Set sessionAffinity: ClientIP on the Service, which keys the conntrack decision on the source IP for a configurable timeout.
Does IPVS mode change how I write manifests? No. It changes the node dataplane only. Your Service and Deployment YAML are identical.
Why did my pod see a NATed source IP?
The default externalTrafficPolicy: Cluster SNATs cross-node traffic. Switch to Local if you need the real client IP.
How do I trace where a packet actually goes?
Start with DNS resolution, then check EndpointSlices for Ready endpoints, then confirm conntrack entries on the node with conntrack -L if the tool is available.
Related
- Why Pods Need Services
- Services Basics
- ClusterIP & Headless
- NodePort & LoadBalancer
- EndpointSlices
- Services Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).