Your .NET service in Kubernetes keeps getting OOMKilled even though the GC heap looks small. How do RAM, virtual memory, working set, private bytes and the GC heap relate, and how do you investigate?

1. The memory map of a containerised .NET process

container limit (cgroup) ---------------------------------------------- 1024 MiB
|  GC heap hard limit (default 75%) ---------------------- 768 MiB     |
|  |  gen0 | gen1 | gen2 | LOH | POH | free space            |        |
|  ---------------------------------------------------------         |
|  native heap (libraries, NativeMemory, unmanaged buffers)           |
|  thread stacks  (threads x up to ~1-1.5 MiB committed as used)      |
|  JIT code, metadata, runtime data structures                        |
|  page cache for files written by the container                      |
----------------------------------------------------------------------
   exceed the cgroup limit -> kernel OOM killer -> exit code 137
   exceed the GC hard limit -> OutOfMemoryException inside the process

2. Counters that tell the story

Rule of thumb: working set − GC committed ≈ native + stacks + runtime. If that gap grows, look outside the GC heap.

3. Managed leak hunting with gcdump

dotnet-gcdump collect -p <pid> -o before.gcdump
# apply load for 10-30 minutes
dotnet-gcdump collect -p <pid> -o after.gcdump
# open both in Visual Studio (or PerfView) and compare: sort by size diff, then follow paths to root

A gcdump triggers a gen 2 GC and is much smaller than a full dump, so it's safer in production.

4. GC configuration knobs

Change one at a time and measure; the defaults are good for most services.

5. Sizing requests and limits

• Measure steady-state and peak working set under realistic load.
• Request ≈ typical working set; limit ≈ peak plus headroom for native memory and bursts.
• Remember the GC treats the limit as a signal: a tiny limit makes it collect more often (more CPU); a huge limit lets the heap grow lazily.
• For many small pods, Workstation GC or DATAS usually gives the best density.

6. Australian context

In regulated industries, memory dumps may contain personal or financial data. Capture them only with approval, store them encrypted in Australian regions, restrict access, and delete them when the investigation closes.

7. Try it yourself

1. Run an ASP.NET Core app in Docker with --memory=256m and watch dotnet-counters while load-testing an endpoint that adds to a static list on every request.
2. Take two gcdumps and find the growing type.
3. Add an endpoint that allocates native memory with NativeMemory.Alloc without freeing it, and watch working set grow while the GC heap stays flat.
4. Compare memory use with Server GC, Server GC with DATAS disabled, and Workstation GC under the same load.

namespace MemoryDiagnostics;

using System.Runtime.InteropServices;

public static class MemoryReport
{
    // A cheap diagnostics endpoint payload (protect it; don't expose publicly).
    public static object Snapshot()
    {
        var gc = GC.GetGCMemoryInfo();
        using var process = System.Diagnostics.Process.GetCurrentProcess();
        return new
        {
            WorkingSetMb = process.WorkingSet64 / 1_048_576,
            PrivateMb = process.PrivateMemorySize64 / 1_048_576,
            GcHeapMb = GC.GetTotalMemory(forceFullCollection: false) / 1_048_576,
            GcCommittedMb = gc.TotalCommittedBytes / 1_048_576,
            GcHardLimitMb = gc.TotalAvailableMemoryBytes / 1_048_576,  // reflects the container limit / hard limit
            Threads = process.Threads.Count,
            ServerGc = System.Runtime.GCSettings.IsServerGC,
        };
    }
}

// Requires <AllowUnsafeBlocks>true</AllowUnsafeBlocks>.
// Native memory is invisible to the GC: if Dispose isn't called, it leaks until the process dies.
public sealed unsafe class NativeBuffer : IDisposable
{
    private void* _pointer;

    public NativeBuffer(nuint size)
    {
        _pointer = NativeMemory.Alloc(size);
        Size = size;
    }

    public nuint Size { get; }

    public void Dispose()
    {
        if (_pointer != null)
        {
            NativeMemory.Free(_pointer);
            _pointer = null;
        }
    }
}

public static class Renderer
{
    private static readonly SemaphoreSlim Concurrency = new(Environment.ProcessorCount);

    public static async Task<byte[]> RenderAsync(Func<NativeBuffer, byte[]> render, CancellationToken ct)
    {
        await Concurrency.WaitAsync(ct);            // bound concurrent native work
        try
        {
            using var buffer = new NativeBuffer(16 * 1024 * 1024);   // freed even if render throws
            return render(buffer);
        }
        finally
        {
            Concurrency.Release();
        }
    }
}

# Live view of the key counters
dotnet-counters monitor -p <pid> --counters System.Runtime[gc-heap-size,gc-committed,working-set,threadpool-thread-count,alloc-rate]

# Confirm an OOM kill
kubectl describe pod <pod> | grep -A3 "Last State"