What is unbounded concurrency, why does it take systems down, and how do you put limits in place in .NET?

1. Little's law, applied

L = λ × W: average in-flight items (L) = arrival rate (λ) × average time in the system (W). It holds for any stable system: a request pipeline, a queue, a connection pool. Use it two ways:

• Sizing: to sustain 500 calls/s to a dependency with 200 ms latency, you need ~100 concurrent calls (plus headroom).
• Failure analysis: a latency increase directly multiplies in-flight work unless a limit caps it.

2. ASP.NET Core rate limiting

builder.Services.AddRateLimiter(o =>
{
    o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    o.OnRejected = (ctx, _) =>
    {
        ctx.HttpContext.Response.Headers.RetryAfter = "5";
        return ValueTask.CompletedTask;
    };

    // Per-client token bucket
    o.AddPolicy("per-client", http => RateLimitPartition.GetTokenBucketLimiter(
        http.User.Identity?.Name ?? http.Connection.RemoteIpAddress?.ToString() ?? "anon",
        _ => new TokenBucketRateLimiterOptions
        {
            TokenLimit = 100, TokensPerPeriod = 50, ReplenishmentPeriod = TimeSpan.FromSeconds(1),
            QueueLimit = 0, AutoReplenishment = true
        }));

    // Global concurrency limit for an expensive endpoint
    o.AddConcurrencyLimiter("reports", l => { l.PermitLimit = 20; l.QueueLimit = 50; });
});

3. Bulkheads for outbound dependencies

builder.Services.AddHttpClient<PaymentsClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler { MaxConnectionsPerServer = 32 })
    .AddResilienceHandler("payments", p =>
    {
        p.AddConcurrencyLimiter(permitLimit: 32, queueLimit: 64);   // bulkhead
        p.AddTimeout(TimeSpan.FromSeconds(2));
        p.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions());
    });

Each dependency gets its own limits, so a slow payments API can't consume all connections, threads or memory needed by everything else.

4. Reject vs queue

A small queue absorbs bursts; a large queue just converts overload into latency and timeouts.

5. Observability

System.Threading.RateLimiting limiters expose statistics (GetStatistics()), and ASP.NET Core's rate limiting emits metrics in .NET 8+.

6. Version notes

7. Try it yourself

1. Call a local endpoint that delays 2 s with Task.WhenAll over 10,000 items; watch memory, sockets (netstat) and errors.
2. Replace with Parallel.ForEachAsync at degree 50 and compare.
3. Add the concurrency limiter to the server; drive it with bombardier -c 200 and count 429s.
4. Set MaxConnectionsPerServer = 8 on the client and observe connection reuse and queueing.
5. Compute the limit you'd need via Little's law for your own service's numbers, then validate with a load test.
6. Add a resilience bulkhead and simulate a slow dependency to confirm other endpoints stay healthy.

using System.Threading.RateLimiting;
using Microsoft.AspNetCore.RateLimiting;

var builder = WebApplication.CreateBuilder(args);

// Inbound limit: at most 20 concurrent report requests, 50 queued, rest rejected with 429.
builder.Services.AddRateLimiter(o =>
{
    o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    o.AddConcurrencyLimiter("reports", l =>
    {
        l.PermitLimit = 20;
        l.QueueLimit = 50;
        l.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
    });
});

// Outbound limit: cap connections to one partner host.
builder.Services.AddHttpClient("partner")
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler { MaxConnectionsPerServer = 32 });

var app = builder.Build();
app.UseRateLimiter();

app.MapPost("/campaigns/{id:int}/send", async (int id, IHttpClientFactory factory, CancellationToken ct) =>
{
    var client = factory.CreateClient("partner");
    int[] recipients = Enumerable.Range(1, 10_000).ToArray();

    // Fan-out limit instead of Task.WhenAll over everything.
    int sent = 0;
    await Parallel.ForEachAsync(recipients,
        new ParallelOptions { MaxDegreeOfParallelism = 32, CancellationToken = ct },
        async (r, token) =>
        {
            await Task.Delay(5, token); // stands in for client.PostAsync(...)
            Interlocked.Increment(ref sent);
        });
    return Results.Ok(new { Campaign = id, Sent = sent });
});

app.MapGet("/reports", async (CancellationToken ct) =>
{
    await Task.Delay(1000, ct);
    return "report";
}).RequireRateLimiting("reports");

app.Run();