How do load balancers work? Compare layer 4 and layer 7 load balancing, explain health checks, algorithms, sticky sessions and connection draining, and why gRPC traffic can end up on only one pod.

1. Where load balancers sit

            DNS / anycast global LB (Front Door, Route 53, Traffic Manager)
                     │
          ┌──────────┴──────────┐
     Region A                Region B
   L7 gateway/ingress      L7 gateway/ingress
          │                       │
   L4 Service (kube-proxy)  L4 Service
     ┌────┼────┐            ┌────┼────┐
    pod  pod  pod          pod  pod  pod

2. Connection vs request balancing

L4, HTTP/1.1 with keep-alive:
  client conn 1 ──► pod A   (all requests on conn 1 → A)
  client conn 2 ──► pod B
  ... many clients, many connections → roughly even

L4, HTTP/2 / gRPC:
  client conn 1 (1,000 concurrent calls) ──► pod A   ← hot
  pod B, pod C idle

L7 HTTP/2 aware (Envoy, Linkerd, YARP):
  client ──► proxy ── call 1 ──► A, call 2 ──► B, call 3 ──► C

The same imbalance appears after scale-out: new pods get no traffic until clients open new connections. PooledConnectionLifetime and gRPC client-side balancing help.

3. Sticky sessions

Affinity (cookie or source IP) pins users to instances. Costs: uneven load, lost sessions when an instance dies, slow scale-in. Prefer stateless instances with state in Redis or a database. Exceptions: SignalR with long polling or server-sent events across multiple servers needs stickiness (or Azure SignalR Service / a backplane).

4. Graceful shutdown and draining

Kubernetes deletes pod
  ├─ endpoint removed from Service (eventually consistent, takes a moment)
  ├─ preStop hook runs (e.g. sleep 5–10 s so LBs stop sending traffic)
  └─ SIGTERM → .NET host stops accepting, finishes in-flight requests
       (HostOptions.ShutdownTimeout, default 30 s)
     terminationGracePeriodSeconds must exceed preStop + shutdown time

Without this, rolling deployments produce a burst of 502/503s.

5. Global load balancing

• DNS-based (Traffic Manager, Route 53): returns different IPs by health, latency or geography; failover is limited by TTLs and client caching.
• Anycast/edge (Front Door, CloudFront, Cloudflare): one IP announced worldwide; the edge routes to the best healthy origin with fast failover.

6. Try it yourself

1. Deploy 3 replicas of a gRPC service; call it from one client with one channel and watch per-pod CPU or request counts.
2. Switch to a headless Service with dns:/// and round_robin; repeat.
3. Remove the preStop hook and run a rolling update under load with k6 or bombardier; count 5xx; then add it back.

// gRPC client-side load balancing across pods (Grpc.Net.Client)
// Requires a headless Kubernetes Service (clusterIP: None) so DNS returns all pod IPs.
var channel = GrpcChannel.ForAddress("dns:///pricing-grpc.pricing.svc.cluster.local:8080", new GrpcChannelOptions
{
    Credentials = ChannelCredentials.Insecure, // TLS terminated by mesh in this example
    ServiceConfig = new ServiceConfig { LoadBalancingConfigs = { new RoundRobinConfig() } },
    HttpHandler = new SocketsHttpHandler
    {
        EnableMultipleHttp2Connections = true,
        PooledConnectionIdleTimeout = Timeout.InfiniteTimeSpan,
        KeepAlivePingDelay = TimeSpan.FromSeconds(60),
        KeepAlivePingTimeout = TimeSpan.FromSeconds(30)
    }
});
var client = new Pricing.PricingClient(channel);

// Separate readiness and liveness for load balancers and Kubernetes
builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddSqlServer(builder.Configuration.GetConnectionString("Orders")!, tags: ["ready"]); // AspNetCore.HealthChecks.SqlServer

builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

app.MapHealthChecks("/health/live", new() { Predicate = r => r.Tags.Contains("live") });
app.MapHealthChecks("/health/ready", new() { Predicate = r => r.Tags.Contains("ready") });

# Pod spec: drain before shutdown
spec:
  terminationGracePeriodSeconds: 40
  containers:
    - name: orders-api
      lifecycle:
        preStop:
          exec: { command: ["sleep", "10"] }   # let endpoints and LBs stop routing first
      readinessProbe: { httpGet: { path: /health/ready, port: 8080 }, periodSeconds: 5 }
      livenessProbe:  { httpGet: { path: /health/live,  port: 8080 }, periodSeconds: 10 }