Deadlock conditions, lock-ordering and async deadlocks, diagnosis with dotnet-dump (syncblk, clrstack, parallelstacks, dumpasync) and prevention.
A deadlock is a cycle where each participant waits for something another holds, so nothing progresses and CPU drops to zero. In .NET the common kinds are lock-ordering deadlocks (thread A holds lock 1 and wants lock 2, thread B the reverse), sync-over-async deadlocks on a SynchronizationContext, non-reentrant SemaphoreSlim re-entry, and tasks that are never completed. I diagnose with a memory dump: dotnet-dump collect, then syncblk to see Monitor owners and waiters, clrstack -all or parallelstacks for thread stacks, and dumpasync for stuck async chains. Fixes are a consistent lock order, shorter locks with no callouts, timeouts, and async all the way.
"Hanging with no CPU tells me nothing is computing: everything is waiting. That's either a deadlock (a cycle of waits) or something waiting on work that will never complete. My first rule is evidence before restart: capture a dump, or at least
dotnet-stackoutput, because a restart destroys the cycle. Then I classify it. Lock deadlocks show blocked threads inMonitor.Enter/Lock.Enter;syncblktells me who owns whichMonitorand who waits, andclrstack -all/parallelstacksshow where each thread is stuck. Async deadlocks often have no blocked threads at all: requests just never complete, so I usedumpasyncto follow awaiting chains to aSemaphoreSlimthat's never released, aTaskCompletionSourcenobody completes, or a sync-over-async wait on aSynchronizationContext. Once I can draw the cycle, I fix it structurally: a single global lock order, no callouts while holding a lock, async all the way, andRunContinuationsAsynchronouslyon promises. Timeouts help turn silent hangs into logged errors, but they don't remove the cycle."
| Condition | Meaning | How we break it |
|---|---|---|
| Mutual exclusion | One holder per resource | Immutable data, single-owner designs (actor/channel) |
| Hold and wait | Holders wait for more | No callouts / nested acquisition under a lock |
| No pre-emption | Can't take a resource away | Timeouts with back-off (TryEnter, WaitAsync(timeout)) |
| Circular wait | A cycle of waits | Global lock order |
Break any one and a deadlock is impossible.
| Kind | Typical cause | What a dump shows |
|---|---|---|
| Lock ordering | Two paths take the same two locks in opposite order | Two threads in Monitor.Enter/Lock.Enter, each owning what the other wants |
| Callback under lock | Event/logging/virtual call while holding a lock | Owner thread inside unexpected code that waits on another lock |
| Sync-over-async with a context | .Result on a UI thread / legacy ASP.NET |
Context thread in Task.Wait; continuation queued to that context |
SemaphoreSlim re-entry |
Nested call waits on the gate it already holds | No blocked threads; dumpasync shows SemaphoreSlim.WaitAsync waiting forever |
| Never-completed task | A TaskCompletionSource not completed on some path |
Awaiting chain ends at an incomplete TCS |
| Inline continuation | SetResult under a lock runs a continuation that needs the lock |
Thread inside SetResult → user code → Monitor.Enter on the same lock |
| Thread-pool "deadlock" (starvation) | All pool threads block waiting for work that needs a pool thread | Many threads in Task.Wait, growing queue (see starvation question) |
| Database | Transactions lock rows in opposite order | SQL Server error 1205 (the DB breaks the cycle by killing one) |
dotnet-dump collect -p <pid> (in Kubernetes: a debug container or dotnet-monitor). Optionally dotnet-stack report -p <pid> twice a few seconds apart.dotnet-dump analyze:| Command | Use |
|---|---|
syncblk |
Owners and waiter counts for contended Monitor locks |
clrstack -all / parallelstacks |
Where every thread is blocked; groups identical stacks |
dumpasync |
Pending async state machines and what they await |
threadpool |
Pool state, queue length |
| Problem | Fix |
|---|---|
| Inconsistent lock order | One documented order (or acquire by sortable id); helper that takes both |
| Callouts under lock | Copy state under the lock; call out after releasing |
| Sync-over-async | Async all the way; ConfigureAwait(false) in libraries |
| Semaphore re-entry | Acquire at public entry points only; helpers assume it's held |
| Uncompleted TCS | Complete on every path; cancellation registrations; timeouts |
| Inline continuation deadlocks | RunContinuationsAsynchronously; complete outside locks |
| Hidden hangs | TryEnter/WaitAsync with timeouts + logging (detection, not a fix) |
Our [inventory service] handled [800 requests/second] and hung completely every [3–4 days]: all requests timed out, CPU near [0%], and a restart "fixed" it. I set up automatic dump capture on hang (a watchdog plus dotnet-monitor). In the next dump, syncblk showed thread 12 owning the _stock lock and waiting for _reservations, and thread 31 owning _reservations and waiting for _stock. A rarely used restock path took the locks in the opposite order. I enforced a single order (stock, then reservations) through a helper that acquires both, removed the second path, added a debug assertion on lock order, and wrote a stress test running both paths concurrently. The hangs never returned.
Checks real debugging ability.
Green flags
Red flags
## 1. Reading `syncblk`
```
> syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
3 0000... 3 1 0000... 2a3c 12 0000... System.Object (_stock)
7 0000... 3 1 0000... 1f80 31 0000... System.Object (_reservations)
```
- `MonitorHeld`: 1 for the owner + 2 per waiter (3 = owned with one waiter).
- Cross-reference: thread 12 owns `_stock`; `clrstack` on thread 12 shows it in `Monitor.Enter(_reservations)`. Thread 31 owns `_reservations` and waits for `_stock`. That's the cycle.
Note: `syncblk` covers **`Monitor`** locks (lock on objects). `System.Threading.Lock` (.NET 9) doesn't use sync blocks, so for those, find threads in `Lock.Enter` via `parallelstacks` and inspect the lock objects' owning thread id fields.
## 2. Async deadlocks: no threads to blame
```
> dumpasync
StateMachine: OrderService+
using System.Threading;
var store = new InventoryStore();
// Reproduce: run both paths at once with the buggy order.
var t1 = Task.Run(() => store.ReserveBuggy("sku-1"));
var t2 = Task.Run(() => store.RestockBuggy("sku-1"));
bool finished = Task.WaitAll([t1, t2], TimeSpan.FromSeconds(2));
Console.WriteLine(finished ? "No deadlock this time" : "Deadlocked: capture a dump now");
public sealed class InventoryStore
{
// Documented order: _stockLock BEFORE _reservationLock.
private readonly Lock _stockLock = new();
private readonly Lock _reservationLock = new();
private readonly Dictionary<string, int> _stock = new() { ["sku-1"] = 10 };
private readonly Dictionary<string, int> _reserved = new();
public void Reserve(string sku)
{
lock (_stockLock)
lock (_reservationLock)
{
_stock[sku]--;
_reserved[sku] = _reserved.GetValueOrDefault(sku) + 1;
}
}
public void ReserveBuggy(string sku)
{
lock (_stockLock)
{
Thread.Sleep(100);
lock (_reservationLock) { _reserved[sku] = _reserved.GetValueOrDefault(sku) + 1; }
}
}
public void RestockBuggy(string sku)
{
lock (_reservationLock) // WRONG order: reservation then stock
{
Thread.Sleep(100);
lock (_stockLock) { _stock[sku] += 5; }
}
}
}
Diagnosis commands:
dotnet-dump collect -p 1234
dotnet-dump analyze core_20261023_101500
> syncblk
> clrstack -all
> parallelstacks
> dumpasync
What happens: the service is restarted to restore traffic.
Why it is wrong: the evidence is gone; the deadlock returns.
Do instead: capture a dump (or at least dotnet-stack) first; automate capture on hang.
What happens: method A locks x then y; method B locks y then x.
Why it is wrong: concurrent calls form a cycle.
Do instead: one documented order, or a helper that acquires in a stable order.
What happens: raising events, logging to a sink, calling virtual methods inside lock.
Why it is wrong: the callee may take other locks or wait on the caller.
Do instead: copy state under the lock; call out after releasing.
What happens: no blocked threads found, so "it's not a deadlock".
Why it is wrong: async deadlocks can involve zero blocked threads.
Do instead: dumpasync / Tasks view to follow awaiting chains.
What happens: TryEnter with timeouts everywhere and retries.
Why it is wrong: the cycle remains; you get intermittent errors and wasted work.
Do instead: use timeouts for detection, then fix the structure.
SemaphoreSlim(1,1)What happens: a public method calls another public method that takes the same gate. Why it is wrong: non-reentrant: self-deadlock. Do instead: acquire once at the entry point; private helpers assume it's held.
TaskCompletionSource completed under a lockWhat happens: SetResult inside lock.
Why it is wrong: inline continuations may need the same lock.
Do instead: RunContinuationsAsynchronously and complete outside the lock.
SynchronizationContextWhat happens: .Result in UI or legacy ASP.NET code.
Why it is wrong: classic context deadlock.
Do instead: async all the way.
syncblk for every lock typeWhat happens: nothing shows up in syncblk for a System.Threading.Lock or SemaphoreSlim deadlock.
Why it is wrong: those don't use sync blocks.
Do instead: use parallelstacks and dumpasync as well.
What happens: the fix ships without a test reproducing the interleaving. Why it is wrong: the same cycle can be reintroduced by a new code path. Do instead: stress tests or Coyote, with timeouts so CI fails instead of hanging.
Deadlocks are design problems that show up as operations problems. My approach is evidence first (dump before restart), then a structural fix that makes the cycle impossible, not a timeout that hides it.
| Evidence | Points to |
|---|---|
syncblk shows two owners waiting on each other |
Lock ordering |
UI or request thread in Task.Wait |
Sync-over-async with a context |
| No blocked threads, stuck tasks | Never-completed task or semaphore re-entry |
| SQL error 1205 | Database lock ordering, add retries and consistent access order |
lock (x) { ... } gives mutual exclusion: only one thread at a time runs the block for the same lock object, and entering and leaving act as memory barriers. For a normal object the compiler emits Monitor.Enter(x, ref lockTaken) in a try/finally with Monitor.Exit. Since .NET 9 and C# 13, if x is a System.Threading.Lock, the compiler instead uses x.EnterScope() and disposes the scope, which is a dedicated, slightly faster lock type. Locks are re-entrant on the same thread, you cannot await inside one (CS1996), and you should lock on a private, readonly object and keep the block short.
A SynchronizationContext is an abstraction for "run this code in the right place", with Post (asynchronous) and Send (synchronous). WinForms, WPF, MAUI and Blazor install one so continuations return to the UI thread or renderer; legacy ASP.NET had one per request; ASP.NET Core and console apps have none. await captures the current context and posts the continuation to it. The classic deadlock: UI code calls .Result on an async method, blocking the only UI thread, while the method's continuation waits to be posted back to that same thread. Neither can proceed.
ThreadPool starvation happens when pool threads are blocked faster than the pool can add new ones, so queued work (requests, continuations, timers) waits. Symptoms: latency rises sharply while CPU stays low, the pool queue length grows, thread count climbs slowly, health checks time out and Kestrel logs heartbeat warnings mentioning possible starvation. I confirm it with dotnet-counters (queue length and thread count), then take dotnet-stack snapshots or a dump to see what the threads are blocked on, usually .Result, .Wait(), synchronous I/O or locks. The fix is to make that code asynchronous; raising SetMinThreads only buys time.
How to tell it (STAR)
Our [inventory service] froze every few days with no CPU. I captured a dump during the hang and `syncblk` showed two threads each holding one lock and waiting for the other. A rarely used restock method took them in reverse order. I enforced one order with a helper and added a stress test, and the hangs never came back.
Last updated 31 Oct 2026.