How do you recognise, diagnose and fix ThreadPool starvation in a production ASP.NET Core service?

1. Why starvation snowballs

request thread T1: client.GetStringAsync(url).Result   -> blocks T1
the HTTP response arrives -> its continuation needs a pool thread to run
all pool threads are blocked in .Result waiting for continuations
-> continuations wait in the queue -> blocked threads keep waiting
-> only slow thread injection (gate thread / hill climbing) breaks the cycle

Each blocked thread waits on work that itself needs a free thread, so throughput collapses until injection catches up. Since .NET 6 the pool compensates faster when threads block on tasks, but it still lags behind a burst.

2. Reading parallelstacks

> parallelstacks
  187 System.Threading.ManualResetEventSlim.Wait
  187 System.Threading.Tasks.Task.SpinThenBlockingWait
  187 System.Threading.Tasks.Task.InternalWaitCore
  187 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
  187 Vendor.Auth.TokenCache.GetToken()        <-- culprit (sync wrapper over async)
  187 Vendor.Auth.AuthHandler.SendAsync
  ...

The frame just above the wait frames is where the blocking was introduced.

3. Kestrel heartbeat warning

Kestrel runs a heartbeat once per second on a timer. When pool threads are starved, the heartbeat itself runs late and Kestrel logs a warning that the heartbeat took longer than expected, noting it could be caused by thread pool starvation. Treat it as an early-warning signal and alert on it.

4. Automating evidence capture

dotnet-monitor collection rules can watch runtime counters and collect stacks/dumps when, for example, queue length exceeds a threshold for 30 seconds:

{
  "CollectionRules": {
    "Starvation": {
      "Trigger": { "Type": "EventCounter", "Settings": {
          "ProviderName": "System.Runtime", "CounterName": "threadpool-queue-length",
          "GreaterThan": 100, "SlidingWindowDuration": "00:00:30" } },
      "Actions": [ { "Type": "CollectStacks" }, { "Type": "CollectDump", "Settings": { "Type": "WithHeap" } } ],
      "Limits": { "ActionCount": 1, "ActionCountSlidingWindowDuration": "01:00:00" }
    }
  }
}

(Check counter names for your runtime version; .NET 9+ also exposes System.Diagnostics.Metrics-based names.)

5. Configuring min threads (temporary)

// runtimeconfig.template.json
{ "configProperties": { "System.Threading.ThreadPool.MinThreads": 200 } }

Or the MSBuild property <ThreadPoolMinThreads>200</ThreadPoolMinThreads>, or ThreadPool.SetMinThreads(200, 200) at startup. Record it as a mitigation with an owner and a removal date.

6. Isolating unavoidable blocking

// Legacy sync-only SDK: run it on a few dedicated threads, never on the pool
var requests = Channel.CreateBounded<(Request Req, TaskCompletionSource<Response> Tcs)>(500);
for (int i = 0; i < 4; i++)
    new Thread(() =>
    {
        foreach (var (req, tcs) in requests.Reader.ReadAllAsync().ToBlockingEnumerable())
        {
            try { tcs.TrySetResult(legacy.Call(req)); }
            catch (Exception ex) { tcs.TrySetException(ex); }
        }
    }) { IsBackground = true, Name = $"legacy-sdk-{i}" }.Start();

Callers await the TCS; only the four dedicated threads ever block.

7. Version notes

8. Try it yourself

1. Run the code example and load /bad with bombardier -c 200 -d 60s; watch dotnet-counters.
2. Load /good with the same settings and compare.
3. Take dotnet-stack report during the /bad test and find the blocking frame.
4. Add System.Threading.ThreadPool.MinThreads = 300 and repeat /bad: better, but memory and threads are much higher.
5. Run in Docker with --cpus=1 and see starvation start at far lower concurrency.
6. Configure a dotnet-monitor rule to capture stacks automatically when queue length is high.

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClient();
var app = builder.Build();

// BAD: sync-over-async blocks a pool thread for the whole call.
app.MapGet("/bad", (IHttpClientFactory f) =>
{
    var client = f.CreateClient();
    string body = client.GetStringAsync("https://example.com/").Result; // starvation under load
    return body.Length;
});

// GOOD: the thread is released while the call is in flight.
app.MapGet("/good", async (IHttpClientFactory f, CancellationToken ct) =>
{
    var client = f.CreateClient();
    string body = await client.GetStringAsync("https://example.com/", ct);
    return body.Length;
});

// Diagnostic endpoint (protect it in real systems).
app.MapGet("/diag/threadpool", () => new
{
    Threads = ThreadPool.ThreadCount,
    Pending = ThreadPool.PendingWorkItemCount,
    Completed = ThreadPool.CompletedWorkItemCount
});

app.Run();

Commands:

dotnet-counters monitor -n MyApi System.Runtime
dotnet-stack report -p 1234 > stacks.txt
dotnet-dump collect -p 1234