A .NET service on Linux starts failing with "Too many open files" and SocketException errors under load. Walk me through how you would diagnose it, including HttpClient socket exhaustion and TIME_WAIT.

1. TCP connection lifecycle and TIME_WAIT

Client (active closer)                 Server
   ESTABLISHED ─── FIN ──────────────────►  CLOSE_WAIT   (server app must close)
   FIN_WAIT_1                               
   FIN_WAIT_2  ◄──────────────── ACK ───
               ◄──────────────── FIN ───   LAST_ACK
   TIME_WAIT   ─── ACK ──────────────────►  CLOSED
      │
      └─ waits 2 × MSL (60 s on Linux) before the port can be reused for
         the same (src ip, src port, dst ip, dst port) tuple

• TIME_WAIT is on the side that closes first. It protects against late packets from the old connection being delivered to a new one. It is normal; it only hurts when you create lots of short connections to the same destination.
• CLOSE_WAIT means the peer closed and our application has not. It never times out by itself, so a growing CLOSE_WAIT count is always an application bug (undisposed response or socket).

2. Ephemeral ports

An outbound connection needs a unique 4-tuple. With the same destination IP and port, only the source port varies.

cat /proc/sys/net/ipv4/ip_local_port_range   # usually 32768 60999 → ~28k ports

At 28,000 ports ÷ 60 s TIME_WAIT ≈ 466 new connections per second to one destination before you run out. Many services exceed that easily if they do not reuse connections.

3. Why new HttpClient() per request hurts

Request ─► new HttpClient ─► new SocketsHttpHandler ─► new pool ─► new TCP (+TLS) ─► dispose ─► TIME_WAIT
Request ─► new HttpClient ─► new SocketsHttpHandler ─► new pool ─► new TCP (+TLS) ─► dispose ─► TIME_WAIT
...

Each HttpClient owns a handler with its own connection pool, so nothing is reused. Compare:

Requests ─► shared HttpClient ─► one pool ─► a few long-lived TCP connections (keep-alive, HTTP/2 multiplexing)

4. The DNS caveat with a static client

A single static HttpClient keeps connections forever by default, so if the target's DNS changes (blue/green, failover), you keep talking to the old IP. Fix with PooledConnectionLifetime, which recycles connections after the given time. IHttpClientFactory handles this for you by rotating handlers (default handler lifetime 2 minutes).

5. Descriptor limits in containers and systemd

.NET raises the soft limit to the hard limit at startup on Linux, so check the hard limit.

6. Other leak sources

• FileSystemWatcher instances (each uses inotify descriptors; also limited by fs.inotify.max_user_instances). Creating one per request or per config reload leaks quickly.
• Process.Start without disposing the Process (pipes for redirected stdio).
• Database connections not returned to the pool (also shows as pool timeout first).
• gRPC channels created per call instead of reused.

7. Try it yourself

1. Write a minimal API that calls https://example.com using new HttpClient() per request.
2. Load it with bombardier -c 50 -d 30s http://localhost:5000/.
3. In another terminal: watch -n1 "ss -tan state time-wait | wc -l" and see thousands.
4. Switch to builder.Services.AddHttpClient("example") and inject IHttpClientFactory. Repeat and watch TIME_WAIT stay low and throughput rise.

// Bad: new client (and new connection pool) per request
app.MapGet("/price/{sku}", async (string sku) =>
{
    using var client = new HttpClient();               // new TCP + TLS every time
    return await client.GetStringAsync($"https://pricing.internal/api/{sku}");
});

// Good: typed client via IHttpClientFactory
builder.Services.AddHttpClient<PricingClient>(client =>
{
    client.BaseAddress = new Uri("https://pricing.internal/");
    client.Timeout = TimeSpan.FromSeconds(5);
});

public sealed class PricingClient(HttpClient http)
{
    public async Task<decimal> GetPriceAsync(string sku, CancellationToken ct)
    {
        using var response = await http.GetAsync($"api/{sku}", ct); // dispose → no CLOSE_WAIT
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<decimal>(ct);
    }
}

// Also good outside DI: one static client with a bounded connection lifetime
private static readonly HttpClient Shared = new(new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(2),   // picks up DNS changes
    PooledConnectionIdleTimeout = TimeSpan.FromMinutes(1),
    MaxConnectionsPerServer = 100
});

# Triage commands
PID=$(pgrep -f OrdersApi)
grep "open files" /proc/$PID/limits
ls /proc/$PID/fd | wc -l
ss -s
ss -tan state time-wait | awk '{print $4}' | sort | uniq -c | sort -rn | head
ss -tanp state close-wait | grep "pid=$PID" | wc -l