A .NET service hangs under load. How do you find a deadlock using the Threads window and Parallel Stacks?

1. First, is it really a deadlock?

Three problems look like "the service hangs" and need different fixes:

Check first with dotnet-counters monitor -p <pid> --counters System.Runtime and look at CPU, thread count, threadpool-queue-length and monitor-lock-contention-count.

2. The deadlock cycle

 Thread 9: Transfer(A -> B)              Thread 12: Transfer(B -> A)
   lock(A)  acquired                       lock(B)  acquired
   lock(B)  waiting ------------+   +------ lock(A)  waiting
                                |   |
                                v   v
                 each holds the lock the other needs
                 --> neither can ever continue

The four Coffman conditions must all hold for a deadlock:

Consistent ordering is the usual fix because it removes the cycle without changing anything else.

3. The local workflow, step by step

 reproduce under the debugger (load test or repro program)
      |
      v
 Debug > Break All (Ctrl+Alt+Break)   -- every thread stops where it is
      |
      v
 Threads window
   group by Location, search "Enter" / "Wait"
   flag suspects: Monitor.Enter, Lock.Enter, SemaphoreSlim.Wait, Task.Wait
      |
      v
 Parallel Stacks (Threads view)
   identical stacks merge: "[2 threads] Transfers.TransferUnsafe"
   hover the box to see which threads are in it
      |
      v
 Parallel Watch: from.Id, to.Id across the flagged threads
   thread 9: from=A to=B     thread 12: from=B to=A
      |
      v
 prove it: Freeze thread 12, Continue --> thread 9 still stuck?
           Thaw 12, freeze 9 ... confirms the cycle

In Visual Studio, the Threads window's Location column shows the top frame for each thread, and threads blocked waiting for a lock show a wait icon. The Parallel Stacks window can also annotate a blocked thread with what it is waiting for, which can point straight at the owning thread.

4. What each window contributes

5. Confirming it in production with a dump

You can't Break All on a production service, so capture a dump of the hung process and analyse it offline:

Then dotnet-dump analyze <file>:

You can also open the dump in Visual Studio and use the same Threads and Parallel Stacks windows as a live session.

6. Monitor locks, System.Threading.Lock and syncblk

lock (obj) on an ordinary object uses Monitor. Contended Monitor ownership is recorded in a sync block, which is why syncblk can list owner threads and waiters.

.NET 9 introduced System.Threading.Lock. When you write lock (myLock) and myLock is a Lock, the compiler uses Lock.EnterScope() instead of Monitor. It doesn't use sync blocks, so syncblk won't list it. The deadlock looks the same in the stacks: threads sitting inside Lock.Enter. Use clrstack -all or Parallel Stacks to find them.

7. Async deadlocks: no thread is blocked

Some hangs have no lock owner to find, because no thread is waiting:

ASP.NET Core has no SynchronizationContext, so sync-over-async there usually shows up as thread pool starvation, not a classic deadlock. For async hangs, use dumpasync and the Tasks view of Parallel Stacks. Always release semaphores in finally.

8. Fixing it by design

9. Observability so you notice sooner

• Request and job timeouts that fail with a clear error.
• A metric for progress or queue age, not just "is the HTTP endpoint alive". A liveness check that answers HTTP can pass while the worker is deadlocked.
• The runtime's lock contention and thread pool queue counters exported to Application Insights or Prometheus.
• A runbook for taking dumps safely: who can take them, where they are stored encrypted, and when they are deleted, because dumps contain customer data.

10. .NET 11 awareness

The diagnosis tools are stable. System.Threading.Lock is increasingly used in libraries, so expect fewer locks to appear in syncblk and rely on stacks. Check the .NET 11 diagnostics release notes for updates to dotnet-dump and dotnet-monitor.

11. Try it yourself

1. Run the code example under the debugger; it deadlocks within seconds.
2. Press Break All, open Threads and Parallel Stacks, and identify which thread holds which lock.
3. Add from.Id and to.Id to Parallel Watch.
4. Freeze one thread and continue to confirm the other stays blocked.
5. Run the deadlocking build outside the debugger, capture a dump with dotnet-dump collect -p <pid>, and find the same stacks with clrstack -all and the owners with syncblk.
6. Change the Sync property to a System.Threading.Lock and see that syncblk no longer lists it but clrstack -all still shows the cycle.
7. Switch to Transfers.Transfer and confirm it completes.

Transfers.cs, a deadlock-prone transfer and the fix. It locks plain objects so the locks are Monitor-based and show up in syncblk:

namespace Payments.Worker;

public sealed class Account(string id, decimal balance)
{
    public string Id { get; } = id;
    public decimal Balance { get; set; } = balance;

    // Monitor-based lock object. With the .NET 9+ System.Threading.Lock type the
    // deadlock is identical, but syncblk will not list it; read the stacks instead.
    public object Sync { get; } = new();
}

public static class Transfers
{
    // Deadlock-prone: one thread locks A then B while another locks B then A.
    public static void TransferUnsafe(Account from, Account to, decimal amount)
    {
        lock (from.Sync)
        {
            Thread.Sleep(10); // demo only: widens the race window so the deadlock reproduces every run
            lock (to.Sync)
            {
                from.Balance -= amount;
                to.Balance += amount;
            }
        }
    }

    // Fix: always acquire locks in a consistent global order.
    public static void Transfer(Account from, Account to, decimal amount)
    {
        var (first, second) = string.CompareOrdinal(from.Id, to.Id) < 0 ? (from, to) : (to, from);
        lock (first.Sync)
        {
            lock (second.Sync)
            {
                from.Balance -= amount;
                to.Balance += amount;
            }
        }
    }
}

Program.cs:

using Payments.Worker;

var a = new Account("ACC-1", 1_000m);
var b = new Account("ACC-2", 1_000m);

var t1 = Task.Run(() => Transfers.TransferUnsafe(a, b, 10m));
var t2 = Task.Run(() => Transfers.TransferUnsafe(b, a, 20m));

if (!Task.WaitAll([t1, t2], TimeSpan.FromSeconds(5)))
{
    Console.WriteLine($"Deadlocked. PID {Environment.ProcessId}: use Break All, or run dotnet-dump collect -p {Environment.ProcessId}");
    Console.ReadLine();
}

Production dump analysis (all commands run inside one analyze session):

dotnet tool install --global dotnet-dump
dotnet-dump collect -p 12345
dotnet-dump analyze ./core_20261022_101500
> clrthreads
> clrstack -all
> syncblk
> dumpasync