Services and Service Discovery: How Pods Find Each Other

A story to begin with

The problem Services solve

Pods are replaceable. Each new pod gets a new IP. If clients had to track pod IPs, every restart would break something. A Service provides:

1. A stable virtual IP and DNS name that never changes.
2. Load balancing across the ready pods behind it.
3. Automatic updates as pods appear and disappear.

flowchart LR
  C[Client pod] -->|http://pricing:80| S[Service pricing<br/>stable IP 10.96.4.12]
  S --> P1[Pod 10.1.2.5 ready]
  S --> P2[Pod 10.1.3.9 ready]
  S -. skipped .-> P3[Pod 10.1.4.2 not ready]
  style S fill:#6366f1,color:#fff
  style P3 fill:#64748b,color:#fff

A Service picks pods using a label selector. The cluster keeps a list of the ready pods matching that selector (the endpoints) up to date automatically. A pod that fails its readiness probe is removed from that list.

apiVersion: v1
kind: Service
metadata: {name: pricing}
spec:
  selector: {app: pricing}         # any ready pod with this label
  ports:
  - name: http
    port: 80                       # the port clients use on the Service
    targetPort: 8080               # the port the container listens on

The Service types

flowchart TB
  subgraph Inside[Inside the cluster]
    CIP[ClusterIP<br/>virtual IP, internal only]
  end
  NP[NodePort<br/>opens a port on every node] --> CIP
  LB[LoadBalancer<br/>asks the cloud for an external LB] --> NP
  EN[ExternalName<br/>DNS alias to an outside name]
  HL[Headless<br/>no virtual IP, DNS returns pod IPs]

For web traffic, most teams use one Ingress or Gateway in front of many ClusterIP Services instead of many load balancers (see the next lesson).

How discovery works: DNS

Every Service gets a DNS name inside the cluster:

<service>.<namespace>.svc.cluster.local
pricing.shop.svc.cluster.local     # full name
pricing.shop                       # works from any namespace
pricing                            # works from the same namespace

sequenceDiagram
  participant App as Checkout pod
  participant DNS as CoreDNS
  participant Svc as Service pricing
  participant Pod as Pricing pod
  App->>DNS: where is pricing?
  DNS-->>App: 10.96.4.12 (the Service IP)
  App->>Svc: HTTP request
  Svc->>Pod: forward to a ready pod
  Pod-->>App: response

Details worth knowing:

• Pods get a /etc/resolv.conf with search domains, so short names work. Setting ndots:5 can cause several failed lookups before a successful one for external names; use fully qualified names with a trailing dot or tune dnsConfig for performance-sensitive apps.
• Pods also receive environment variables for Services that existed when they started, but DNS is the right mechanism; do not rely on the variables.
• kube-proxy (or an eBPF dataplane) turns Service IPs into pod IPs using iptables, IPVS or eBPF rules on each node.

Debugging "my Service does not work"

Work from the inside out. Most failures are one of four things.

flowchart TD
  A[Service not reachable] --> B{kubectl get endpoints svc<br/>shows addresses?}
  B -- none --> C[Selector does not match pod labels,<br/>or pods are not Ready]
  B -- yes --> D{Can you reach a pod IP:port directly?}
  D -- no --> E[App not listening on that port,<br/>wrong targetPort, container crashed]
  D -- yes --> F{Does DNS resolve the name?}
  F -- no --> G[CoreDNS problem or wrong name or namespace]
  F -- yes --> H[NetworkPolicy blocking, or mesh policy]
  style C fill:#f87171,color:#fff

kubectl get svc pricing -o wide
kubectl get endpoints pricing                 # empty = selector or readiness problem
kubectl get pods --show-labels                # compare labels with the selector
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash
# inside the debug pod:
nslookup pricing.shop.svc.cluster.local
curl -v http://pricing.shop/healthz
curl -v http://10.1.2.5:8080/healthz          # a pod IP, bypassing the Service

The pattern: endpoints empty means selector or readiness; pod IP works but Service does not means port mapping or the dataplane; name does not resolve means DNS.

Useful options

Standardising service networking in your organization

flowchart LR
  A[Naming standard:<br/>service names, ports, labels] --> B[Service template<br/>in the base chart]
  B --> C[Timeouts and retries<br/>in clients or mesh]
  C --> D[Documented debug runbook<br/>with netshoot pod]
  D --> E[Dashboards: endpoints count,<br/>DNS latency and errors]
  style B fill:#6366f1,color:#fff

1. Naming: lowercase, one Service per app, named ports, consistent app and team labels, DNS names documented in a service catalogue.
2. Template: the base chart creates the Service, with targetPort tied to the container port name so they cannot drift.
3. Never hard-code IPs; add a lint check in CI that rejects pod or node IPs in configuration.
4. Client behaviour: set connect and request timeouts, retries with backoff, and connection reuse; cluster networking is reliable but not instant.
5. Observability: alert on Services with zero endpoints, on CoreDNS latency and error rate, and on 5xx from the entry point.
6. Runbook: publish the debug method above and the netshoot command. Practise it once in a training session.
7. Cost: limit LoadBalancer Services via policy; route external traffic through shared Ingress or Gateway.

Pitfalls

• A selector that matches pods of several apps by accident.
• Forgetting that readiness gates endpoints: a Service with no ready pods has no endpoints.
• One cloud LoadBalancer per Service, quickly becoming a big bill.
• Assuming DNS is free: heavy lookup traffic needs CoreDNS capacity or NodeLocal DNSCache.

Interview questions

Q1. Why do we need Services? Pod IPs are ephemeral. A Service provides a stable virtual IP and DNS name and load balances across the current ready pods, updating automatically as pods change.

Q2. ClusterIP vs NodePort vs LoadBalancer? ClusterIP is internal only; NodePort additionally opens a port on every node; LoadBalancer provisions an external cloud load balancer on top of those.

Q3. What are endpoints and how do they relate to readiness? Endpoints (EndpointSlices) list the IP and port of pods selected by a Service. Only Ready pods are included, so a failing readiness probe removes a pod from traffic.

Q4. How does a pod find another service by name? Through cluster DNS (CoreDNS): service.namespace.svc.cluster.local, with search domains that allow short names within the namespace.

Q5. A Service has no endpoints. What are the likely causes? The selector does not match any pod labels, or pods exist but are not Ready (probe failing), or the pods are in a different namespace.

Q6. What is a headless Service and why use it? A Service with clusterIP: None that skips the virtual IP and returns the pods' IPs in DNS, giving each pod its own record. It is used with StatefulSets and clients that must address individual pods.

Next: unlock Ingress and Gateway API for routing external traffic into the cluster.