How Services Find Each Other
Summary
- Service discovery in Kubernetes is the mechanism that lets one pod reach another workload by a stable name instead of a mutable pod IP.
- Insight: A Service is a stable virtual IP plus a DNS name in front of an ever-changing set of pod endpoints.
- Key Concepts: ClusterIP, Endpoints/EndpointSlice, kube-proxy (or eBPF/CNI dataplane), CoreDNS, and the
svc.cluster.localsearch domain. - When to Use: Every in-cluster call between microservices should target a Service name, never a hardcoded pod IP.
- Limitations/Trade-offs: DNS caching, TTLs, and headless Services change the resolution behavior you get.
- Related Topics: DNS record formats, CoreDNS configuration, and reaching services outside the cluster.
Foundations
Pods are ephemeral. They are created, rescheduled, and deleted constantly, and each one gets a fresh IP.
You cannot build a reliable system on pod IPs that vanish on the next rollout.
A Service solves this by giving a group of pods one stable identity.
The Service selects pods with a label selector, and the control plane keeps a live list of the healthy backends behind it.
That stable identity has two faces: a virtual IP (the ClusterIP) and a DNS name.
The DNS name is what application code should use, because it survives Service recreation and is human-readable.
The canonical name is service.namespace.svc.cluster.local.
Read it right to left: cluster.local is the cluster zone, svc marks it as a Service record, namespace scopes it, and the leftmost label is the Service name.
Mechanics & Interactions
Three subsystems cooperate to turn a name into a working connection.
First, the control plane tracks backends.
When pods matching a Service's selector become Ready, their addresses are written into EndpointSlice objects associated with that Service.
EndpointSlice is the modern, shardable replacement for the older single Endpoints object and scales to large backend sets.
Second, CoreDNS answers the name lookup.
CoreDNS runs as a Deployment in kube-system, fronted by a Service usually named kube-dns for backward compatibility.
Every pod's /etc/resolv.conf points nameserver at that DNS Service's ClusterIP, injected by the kubelet.
When your code resolves payments, the stub resolver sends the query to CoreDNS, which returns the ClusterIP of the payments Service.
Third, the dataplane forwards packets.
The ClusterIP is virtual; nothing listens on it directly.
kube-proxy (in iptables or IPVS mode) or an eBPF-based CNI like Cilium programs each node so that traffic to the ClusterIP is load-balanced across the current Ready endpoints.
So the DNS answer gets you an IP, and the node's dataplane rewrites that to a real pod.
The search domain is what makes short names work.
A pod's resolv.conf includes a search line such as myns.svc.cluster.local svc.cluster.local cluster.local.
Resolving the bare name payments triggers the resolver to try each suffix in turn, so payments expands to payments.myns.svc.cluster.local within the same namespace.
That is why a short name resolves in-namespace but you need the fuller name to reach another namespace.
An ndots:5 option in resolv.conf controls when the resolver tries the search list before treating a name as absolute.
Advanced Considerations & Applications
Normal Services (with a ClusterIP) give you an A/AAAA record pointing at the single virtual IP.
Headless Services (clusterIP: None) are different: DNS returns the individual pod IPs directly, one A record per Ready endpoint.
Headless mode is how StatefulSets give each pod a stable per-pod DNS name like pod-0.myservice.myns.svc.cluster.local, which databases and quorum systems rely on.
Caching is the subtle part.
CoreDNS caches answers, and many language runtimes and libc also cache DNS.
If a Service's ClusterIP is stable (the common case), stale caching rarely hurts because the IP does not change even as pods churn.
Headless Services are more sensitive, since the returned pod IPs do change, so long client-side caches can point at dead pods.
For that reason, prefer ClusterIP Services for load-balanced traffic and reserve headless for cases that genuinely need per-pod addressing.
publishNotReadyAddresses: true on a headless Service exposes pods before they pass readiness, which some peer-discovery protocols need during bootstrap.
Common Misconceptions
"DNS load-balances my traffic."
It does not for a normal Service; DNS returns one ClusterIP and the node's dataplane does the balancing.
"Pods talk to each other by pod IP."
They can, but you should target Service names so the connection survives rescheduling.
"CoreDNS is a full external resolver."
CoreDNS authoritatively serves cluster names and forwards everything else upstream; it is not your internet DNS by itself.
"The Service name is enough from any namespace."
A bare name resolves only within the caller's namespace; cross-namespace calls need service.namespace or the FQDN.
"kube-proxy resolves DNS."
kube-proxy programs the dataplane for ClusterIPs; name resolution is CoreDNS's job.
FAQs
What is the shortest name I can use?
Inside the same namespace, just the Service name, for example payments.
How do I reach a Service in another namespace?
Use payments.billing or the full payments.billing.svc.cluster.local.
Why do some Services return multiple IPs?
They are headless Services (clusterIP: None), which publish per-pod A records instead of one virtual IP.
Where does a pod learn the DNS server address?
The kubelet writes it into the pod's /etc/resolv.conf as the cluster DNS Service ClusterIP.
Does the ClusterIP change on redeploy?
No; the ClusterIP is stable for the life of the Service object, even as backing pods change.
What handles the actual packet forwarding?
kube-proxy or an eBPF CNI dataplane rewrites ClusterIP traffic to a Ready endpoint.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).