Load Balancing

TL;DR: A load balancer spreads incoming requests across a pool of backend instances so no one server is overloaded, and pulls failed instances out of rotation via health checks. L4 balancers route on IP/port (fast, protocol-agnostic, no request inspection); L7 balancers read the HTTP request (path, headers, cookies) and route by content, enabling path-based routing, sticky sessions, and per-tenant rules. The distribution algorithm (round-robin, least-connections, consistent-hash) matters most when request costs are uneven, which is exactly the case for LLM inference.

What a load balancer actually does

Two jobs. First, distribution: take a stream of requests and fan them across N healthy backends so capacity is used evenly. Second, failure isolation: continuously probe backends and stop sending traffic to ones that are down or unhealthy, so a single dead instance becomes invisible to users instead of a stream of errors. The second job is what makes a load balancer a fault-tolerance building block, not just a traffic sprayer.

L4 vs L7

• L4 (transport layer). Routes on IP address and TCP/UDP port without looking inside the request. It is fast and cheap (just forwarding packets/connections) and works for any protocol, but it cannot make content-aware decisions. Good for raw throughput and non-HTTP traffic.
• L7 (application layer). Terminates the connection and reads the HTTP request, so it can route by path (/v1/chat to the GPU pool, /v1/embed to the CPU pool), by header or cookie (sticky sessions, tenant routing), do TLS termination, and apply per-route rate limits. More work per request, far more control. Most API gateways and ingress controllers are L7.

Distribution algorithms

For LLM inference this is not academic. A round-robin balancer treats a 5-token request and a 4000-token streaming generation identically, so it piles long jobs onto a few unlucky instances while others idle. Least-connections (or least-outstanding-requests) tracks how busy each backend actually is and routes around the slow ones, which matches the highly variable cost of generation far better. Consistent-hash routing (consistent hashing) sends the same key to the same backend, which is how you get session affinity or KV-cache and prefix-cache hits on the same instance.

Health checks

A health check is a periodic probe (an HTTP GET /healthz, a TCP connect) that decides whether a backend stays in rotation. Active checks poll on an interval; passive checks watch real traffic and eject a backend after a burst of errors. The subtlety: a shallow check (is the port open?) misses a process that is up but wedged, while a deep check (can it actually serve?) catches more but costs more and risks flapping. Tune the threshold so one slow probe does not eject a healthy node, and make sure the check reflects readiness (a node warming a model is not ready even though it is alive).

Why interviewers probe this

Every horizontally scaled service sits behind a load balancer, so "how do requests reach your backends, and what happens when one dies" is a baseline design question. The L4/L7 distinction shows you know where routing decisions live and what they cost. The algorithm choice is where applied-AI candidates separate themselves: naming least-connections for uneven inference costs, or consistent-hash for cache affinity, shows you understand your workload rather than reciting "round-robin." Health checks are the answer to the fault-tolerance follow-up.

Common misconceptions

• "Round-robin is always fine." It assumes equal request cost; for LLM generation with wildly uneven durations, least-connections or least-outstanding-requests balances far better.
• "L7 is just a better L4." L7 inspects the request and costs more per call; L4 is faster and protocol-agnostic. You pick by whether you need content-aware routing.
• "A load balancer makes you fault-tolerant for free." Only with health checks that actually reflect readiness; a shallow check keeps routing to a wedged instance.
• "Sticky sessions are mandatory." They hurt balance and complicate failover; prefer stateless backends and reach for affinity only when you need cache locality or in-memory session state.
• "The load balancer is never the bottleneck." It has connection and throughput limits too; very large fleets need multiple LBs or DNS/anycast load spreading in front.

Key takeaways

• A load balancer distributes requests across healthy backends and removes failed ones via health checks.
• L4 routes on IP/port (fast, protocol-agnostic); L7 reads the request and routes by path, header, or cookie.
• Round-robin assumes equal cost; least-connections handles uneven request durations like LLM generation; consistent-hash gives sticky, cache-friendly routing.
• Health checks must reflect real readiness (a model still warming is not ready), with thresholds tuned to avoid flapping.
• Prefer stateless backends; use session affinity only when cache locality or in-memory state demands it.