Latency, bandwidth and throughput defined; RTT, bandwidth-delay product, Little's law and tail latency; why chatty calls across regions hurt and how batching, caching and placement fix it.
For most APIs, latency × number of round trips dominates, not bandwidth. A request that makes 50 sequential database calls at 2 ms RTT spends 100 ms on the network; move the database to another region at 30 ms RTT and it spends 1.5 s. Fix it by reducing round trips (batching, joins, fewer chatty calls), moving compute near data, caching, and parallelising independent calls.
| Term | Highway analogy | Unit | Example |
|---|---|---|---|
| Latency | Time for one car to drive the road | ms | 15 ms Sydney–Melbourne RTT |
| Bandwidth | Number of lanes | Gbit/s | 10 Gbit/s link |
| Throughput | Cars actually arriving per minute | req/s, MB/s | 4,000 req/s |
Adding lanes (bandwidth) does not make one car arrive sooner (latency).
| Path | RTT |
|---|---|
| Same host (loopback) | < 0.1 ms |
| Same availability zone | ~0.2–0.5 ms |
| Across zones in a region | ~1–2 ms |
| Sydney ↔ Melbourne | ~12–15 ms |
| Sydney ↔ Singapore | ~90 ms |
| Sydney ↔ US West | ~140–160 ms |
Chatty (N+1): 1 query for orders + 100 queries for lines
101 round trips × 30 ms = 3,030 ms
Chunky: 1 query with a join (or 2 queries with IN)
2 round trips × 30 ms = 60 ms
A company moved its SQL database to a different Azure region for cost reasons while the API stayed put. RTT rose from 1 ms to 28 ms. An order page that ran about 80 EF Core queries (lazy loading in a loop) went from 120 ms to 2.4 s. Bandwidth was irrelevant. We co-located the database again, then fixed the N+1 with Include and projections (3 queries), and added a short-lived cache for reference data. The page went to 40 ms even in the worst case.
Checks performance reasoning about networks, especially the difference between latency and bandwidth and the cost of round trips.
Green flags
Red flags
total latency = propagation (distance)
+ transmission (size / bandwidth)
+ queuing (busy routers, busy servers, thread pool queues)
+ processing (your code, the database)
Propagation is physics: ~5 µs per km in fibre, so Sydney–London (~17,000 km by cable) is at least ~170 ms one way. No code can fix that; only moving things closer can.
The amount of data "in flight" needed to fill a link:
BDP = bandwidth × RTT
1 Gbit/s × 150 ms = 150 Mbit ≈ 18.75 MB in flight
A single TCP connection can only send one window per round trip, so:
max throughput per connection ≈ window / RTT
64 KB window / 150 ms ≈ 0.43 MB/s (no matter how fat the pipe is)
Modern OSes auto-tune windows, but proxies, old configurations or small application buffers can still cap long-distance transfers. Parallel streams or larger windows help.
concurrency (L) = throughput (λ) × latency (W)
To serve 2,000 req/s with 50 ms latency you need ~100 requests in flight. If each request blocks a thread, you need ~100 threads; with async I/O, threads are freed while waiting. If a downstream call's latency doubles, the concurrency you need doubles too, which is how slow dependencies exhaust connection pools and thread pools.
Averages hide pain. If a page fans out to 10 services and each has a 1% chance of a slow response, about 10% of page loads hit at least one slow call. Track p95/p99/p99.9, use timeouts, and consider hedged requests for critical reads.
| Problem | Technique |
|---|---|
| Many sequential calls | Batch APIs, joins, IN queries, GraphQL/BFF aggregation |
| Independent calls in sequence | Run them concurrently (Task.WhenAll) |
| Repeated reads | Cache (in-memory, distributed, HTTP caching, CDN) |
| Far-away users | CDN, edge, regional deployments |
| Large payloads | Compression (Brotli/gzip), pagination, projections |
| Handshake costs | Connection reuse, HTTP/2, TLS 1.3 |
tc qdisc add dev eth0 root netem delay 30ms in a Linux test VM (or Toxiproxy) to add latency between your API and database.awaits for three independent HTTP calls with Task.WhenAll and compare.// Chatty: N+1 queries, each paying a round trip
var orders = await db.Orders.Where(o => o.CustomerId == id).ToListAsync(ct);
foreach (var order in orders)
{
order.Lines = await db.OrderLines.Where(l => l.OrderId == order.Id).ToListAsync(ct); // 1 RTT each
}
// Chunky: one round trip with a projection
var summaries = await db.Orders
.Where(o => o.CustomerId == id)
.Select(o => new OrderSummary(o.Id, o.PlacedAt, o.Lines.Sum(l => l.Quantity * l.UnitPrice)))
.ToListAsync(ct);
// Independent downstream calls: run concurrently instead of sequentially
// Sequential: ~ (40 + 60 + 30) ms = 130 ms
// Concurrent: ~ max(40, 60, 30) ms = 60 ms
var profileTask = profiles.GetAsync(userId, ct);
var ordersTask = orderClient.GetRecentAsync(userId, ct);
var loyaltyTask = loyalty.GetPointsAsync(userId, ct);
await Task.WhenAll(profileTask, ordersTask, loyaltyTask);
return new Dashboard(profileTask.Result, ordersTask.Result, loyaltyTask.Result);
Mistake: Using "bandwidth" and "speed" interchangeably. Why it is wrong: A bigger pipe does not reduce round-trip time, and most API payloads are tiny. Instead: Reason about latency times round trips for request/response work, and bandwidth for bulk transfers.
Mistake: Moving a dependency to another region without checking chattiness. Why it is wrong: Every sequential call now pays tens of milliseconds, so an 80-query page goes from 120 ms to seconds. Instead: Co-locate chatty components, or redesign to fewer, larger calls before moving.
Mistake: Reporting average latency. Why it is wrong: Averages hide the tail that users feel, especially when one page fans out to many services. Instead: Track p95 and p99 and set SLOs on percentiles.
Mistake: Awaiting independent calls one after another.
Why it is wrong: Their latencies add up for no reason.
Instead: Start them together and await Task.WhenAll, with timeouts and cancellation.
Mistake: Blocking threads on I/O (.Result, synchronous database calls) under load.
Why it is wrong: By Little's law, higher latency needs more concurrency, and blocked threads starve the thread pool long before the CPU is busy.
Instead: Use async I/O end to end, and limit concurrency to slow dependencies with bulkheads.
I start performance conversations by asking how many round trips, and how far. Physics sets a floor that code cannot beat, so placement, data locality and API granularity matter more than micro-optimisation.
| Pattern | Reduces | Trade-off |
|---|---|---|
| Co-locate app and database in one region or zone | RTT per call | Cost, availability design |
| Projections, joins, batch endpoints | Number of round trips | Larger, more specific queries |
| Backend-for-frontend aggregation | Client round trips over slow links | Another service to own |
Task.WhenAll for independent calls |
Sum of latencies to the maximum | More concurrent load downstream |
| Caching (memory, Redis, HTTP, CDN) | Repeated round trips | Staleness and invalidation |
| Async I/O end to end | Threads needed per request | Requires async all the way down |
| Timeouts, bulkheads, rate limits | Blast radius of slow dependencies | Some requests fail fast by design |
I watch these proactively: OpenTelemetry traces show round trips per request, dependency p99 and query counts, and I add a query-count budget test in CI plus latency injection (Toxiproxy or tc netem) in performance tests, so chatty designs are caught before a region move exposes them.
api.example.com to an IP address (OS cache, then a resolver).api.example.com (TLS 1.3: one more round trip).GET /orders/42 with headers; often a load balancer or reverse proxy terminates TLS and forwards to an app instance.HttpClient keeps alive and reuses for the next request, skipping steps 1–4.On a cold connection, DNS + TCP + TLS can cost more than the request itself, which is why connection reuse matters.
A load balancer spreads traffic across healthy instances.
Health checks remove unhealthy instances (in Kubernetes: readiness for traffic, liveness for restarts). Algorithms: round robin, least connections/requests, hashing. Sticky sessions pin a user to one instance, which hurts scaling and failover. Connection draining stops new traffic to an instance before shutdown so in-flight requests finish.
gRPC uses long-lived HTTP/2 connections, so an L4 balancer balances connections, not calls: one client connection carries all its calls to one pod. Use L7 balancing (mesh/Envoy) or client-side load balancing.
CPUs load memory in cache lines of 64 bytes, so after one access the neighbouring bytes are already in cache (spatial locality), and recently used data is likely still there (temporal locality). Hardware prefetchers also spot sequential access and fetch ahead. A C# int[,] is stored row-major: one row's elements are next to each other. Walking row by row uses every byte of each line and the prefetcher keeps up; walking column by column jumps a whole row each step, wasting most of each line and missing cache, often 5–10× slower for large arrays. The same logic favours arrays of structs over lists of objects scattered on the heap, and struct-of-arrays layouts for hot numeric fields.
How to tell it (STAR)
**S:** After our database moved to another region, the [customer portal] order page went from [120 ms] to [2.4 s]. **T:** I had to restore performance and stop it recurring. **A:** Telemetry showed [80] queries per page, and each one now paid a [28 ms] round trip. The real issue was an N+1 pattern that the extra latency exposed. We moved the database back, rewrote the queries as projections and cached reference data, and added a query-count budget test. **R:** The page dropped to [40 ms], and the budget test has caught [several] N+1 regressions since.
Last updated 23 Oct 2026.