Explain the memory hierarchy: registers, L1, L2 and L3 caches, and RAM. How big and how fast is each level, and why does it matter for .NET code?

1. The pyramid

                 [registers]          < 1 ns       bytes
                [  L1 cache  ]         ~1 ns       32-64 KB per core
              [    L2 cache    ]       ~3-4 ns     1-2 MB per core
          [        L3 cache        ]   ~10-20 ns   tens of MB, shared
     [               RAM                ]  ~60-100 ns   GBs
 [                  SSD / network                    ]  micro- to milliseconds

2. Scaling latency to human time

If an L1 hit took 1 second:

That's why one database call costs more than millions of in-memory operations, and why in-memory hot loops care about cache misses.

3. How a cache is organised

• Memory is split into 64-byte lines. Each cache holds a fixed number of lines.
• Caches are set-associative: an address maps to a small set (for example 8 or 12 "ways"), and the cache picks a victim in that set to evict (roughly least recently used).
• Inclusive/exclusive policies differ between vendors (whether L3 also holds what's in L2), which affects effective capacity.
• Writes go to the cache first and are written back to RAM later (write-back).
• Separate L1i (instructions) and L1d (data) caches: large amounts of code (big methods, many generic instantiations) can also miss.

4. The TLB

• Programs use virtual addresses; the CPU translates them to physical addresses using page tables.
• The TLB caches recent translations (a few thousand entries). With 4 KB pages, that covers only a few MB to tens of MB.
• Random access over a huge heap misses the TLB as well as the cache, adding a page-table walk. Large pages can help specialised workloads (the GC has a large-pages option for very large heaps on servers).

5. Shared caches and noisy neighbours

L3 is shared by cores, and in the cloud by other VMs or containers on the same host. A neighbour streaming through memory can evict your data, which is one reason latency varies between identical instances.

6. Rules of thumb for .NET

7. Try it yourself

1. Write a benchmark that walks an array with a random permutation of indexes (pointer-chasing style) for sizes 16 KB, 256 KB, 4 MB, 64 MB and 512 MB. Plot nanoseconds per access and spot the steps at L1, L2, L3 and RAM.
2. Run lscpu (Linux) or CPU-Z (Windows) to find your cache sizes and compare with the plot.
3. Repeat with sequential access and explain why the steps almost disappear (prefetching).

namespace MemoryHierarchy;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

// Measures average latency per dependent load at different working-set sizes.
public class LatencyLadder
{
    [Params(16 * 1024, 256 * 1024, 4 * 1024 * 1024, 64 * 1024 * 1024)]
    public int Bytes;

    private int[] _next = [];
    private const int Steps = 10_000_000;

    [GlobalSetup]
    public void Setup()
    {
        int count = Bytes / sizeof(int);
        // Build one random cycle through all slots so each load depends on the previous one.
        var order = Enumerable.Range(0, count).ToArray();
        new Random(7).Shuffle(order);
        _next = new int[count];
        for (var i = 0; i < count; i++)
        {
            _next[order[i]] = order[(i + 1) % count];
        }
    }

    [Benchmark(OperationsPerInvoke = Steps)]
    public int ChaseRandom()
    {
        int index = 0;
        for (var i = 0; i < Steps; i++)
        {
            index = _next[index];          // next address unknown until this load completes
        }
        return index;
    }

    [Benchmark(OperationsPerInvoke = Steps)]
    public int WalkSequential()
    {
        int sum = 0, length = _next.Length;
        for (var i = 0; i < Steps; i++)
        {
            sum += _next[i % length];      // predictable, prefetcher-friendly
        }
        return sum;
    }
}

public static class Program
{
    public static void Main() => BenchmarkRunner.Run<LatencyLadder>();
}

On typical hardware, ChaseRandom reports about 1 ns per step at 16 KB, a few ns at 256 KB, 10–20 ns at 4 MB and 70–100+ ns at 64 MB, while WalkSequential stays low throughout.