How do you investigate a memory leak in a .NET service in production, step by step?

1. Choosing the right tool

2. gcdump vs full dump

Use gcdumps to find what grows; a full dump to find why it's alive and inspect values.

3. Reading a gcroot chain

> gcroot 00007f8a1c2d4e80
HandleTable:
    00007f8a0bf513f8 (strong handle)
    -> 00007f8a1a001018 System.Object[]                 (static field storage)
    -> 00007f8a1a2c3b40 MyApp.ConnectionMonitor
    -> 00007f8a1c9e0a10 System.EventHandler<Boolean>    (multicast delegate)
    -> 00007f8a1c9a8f20 System.Object[]                 (invocation list)
    -> 00007f8a1c2d4e60 System.EventHandler<Boolean>
    -> 00007f8a1c2d4e80 MyApp.NotificationContext        <- leaked object

Root types you'll see: strong handles (statics, GCHandle), stack (a thread holding it right now), pinned handles, finaliser queue, async state machines referenced from timers/tasks. A stack root on a thread that runs forever (background loop) is a leak too.

4. Useful SOS commands

5. Native leak investigation

1. Compare eeheap -gc (managed) with total process memory (maddress -summary or OS tools).
2. Linux: pmap -x <pid>, /proc/<pid>/smaps; Windows: VMMap, RAMMap.
3. Look for growth in anonymous native regions; correlate with native libraries (compression, image processing, SQL native drivers, gRPC).
4. Check that every SafeHandle owner is disposed; check thread count.
5. For glibc allocator fragmentation in many-threaded apps, consider MALLOC_ARENA_MAX.

6. Kubernetes practicalities

• Collecting: kubectl debug -it <pod> --image=<image-with-dotnet-tools> --target=<container> to share the process namespace, or run dotnet-monitor as a sidecar sharing the diagnostic socket (/tmp).
• Disk space: a 4 GB process produces a ~4 GB dump; mount a volume.
• OOMKilled leaves no dump: the kernel sends SIGKILL, so crash-dump settings don't fire. Use dotnet-monitor collection rules on a memory threshold (e.g. GC heap > 80% of limit) to capture data before the kill.
• Crash dumps for managed crashes: DOTNET_DbgEnableMiniDump=1, DOTNET_DbgMiniDumpType, DOTNET_DbgMiniDumpName.
• Security: dumps contain secrets and PII; handle and delete them accordingly.

7. Version notes

8. Try it yourself

1. Run the code example; watch subscriber count and heap grow each round.
2. Collect two gcdumps between rounds and diff them in Visual Studio or PerfView.
3. Collect a full dump; run dumpheap -stat -type NotificationContext, dumpheap -mt, then gcroot on one instance.
4. Fix the leak with using var context = ... and confirm the subscriber count stays at zero.
5. Simulate a native leak with NativeMemory.Alloc in a loop and compare dotnet-counters GC heap vs working set.
6. Run the leaky app in a container with a memory limit and dotnet-monitor configured with a memory-threshold trigger to collect a gcdump automatically.

// Deliberate leak for practice: scoped objects subscribe to a singleton event and never unsubscribe
var monitor = new ConnectionMonitor();

for (int round = 1; round <= 20; round++)
{
    for (int i = 0; i < 10_000; i++)
    {
        var context = new NotificationContext(monitor);   // simulates a per-request object
        context.Process();
        // BUG: context is never disposed, so it stays subscribed
    }

    Console.WriteLine($"Round {round}: GC heap {GC.GetTotalMemory(forceFullCollection: true) / 1024 / 1024} MB, " +
                      $"subscribers {monitor.SubscriberCount}");
    await Task.Delay(500);
}

Console.WriteLine($"PID {Environment.ProcessId}: collect with dotnet-gcdump / dotnet-dump now");
Console.ReadLine();

public sealed class ConnectionMonitor
{
    public event EventHandler<bool>? StatusChanged;
    public int SubscriberCount => StatusChanged?.GetInvocationList().Length ?? 0;
    public void Raise(bool up) => StatusChanged?.Invoke(this, up);
}

public sealed class NotificationContext : IDisposable
{
    private readonly ConnectionMonitor _monitor;
    private readonly byte[] _state = new byte[1024];   // makes the leak visible

    public NotificationContext(ConnectionMonitor monitor)
    {
        _monitor = monitor;
        _monitor.StatusChanged += OnStatusChanged;      // singleton now references this object
    }

    public void Process() => _state[0]++;

    private void OnStatusChanged(object? sender, bool up) => _state[1] = up ? (byte)1 : (byte)0;

    public void Dispose() => _monitor.StatusChanged -= OnStatusChanged;   // the fix: dispose each context
}

dotnet-counters monitor -p <pid> --counters System.Runtime
dotnet-gcdump collect -p <pid> -o before.gcdump
# ...wait, then
dotnet-gcdump collect -p <pid> -o after.gcdump
dotnet-dump collect -p <pid> -o leak.dmp
dotnet-dump analyze leak.dmp
> dumpheap -stat -type NotificationContext
> dumpheap -mt <MethodTable>
> gcroot <address>