What is a cache miss? How would you find out whether cache misses are slowing down a .NET service, and how do you reduce them?

1. What happens on a miss

load [addr]
   |
   L1? --hit--> ~4 cycles
   | miss
   L2? --hit--> ~12-15 cycles
   | miss
   L3? --hit--> ~40-70 cycles
   | miss
   TLB miss? -> page-table walk (more memory accesses)
   |
   RAM  ------> ~200-300 cycles, line filled into L3, L2, L1

Out-of-order execution hides some latency by working on independent instructions, but a chain of dependent loads (pointer chasing) exposes the full cost on every hop.

2. Pointer chasing versus streaming

List<Order> -> Order -> Customer -> Address -> Country      four dependent misses per item

OrderRow[] { CustomerId, CountryCode, Total }               one sequential stream

The first can't be prefetched because each address depends on the previous load. The second streams at memory bandwidth.

3. Reading perf output

Use ratios, not raw counts, and compare before and after the same load test.

4. Getting symbol names for JIT code on Linux

• DOTNET_PerfMapEnabled=1 writes /tmp/perf-<pid>.map so perf report can name JIT-compiled methods.
• Alternatively, dotnet-trace with the default CPU profile gives managed stacks without hardware counters; combine both views.

5. Hot/cold splitting example

// Before: one class, 20 fields, read 3 in the hot path
public sealed class Product { public int Id; public decimal Price; public bool InStock; /* ...17 more... */ }

// After: hot data compact and contiguous, cold data looked up when needed
public readonly record struct ProductHot(int Id, decimal Price, bool InStock);
ProductHot[] hot;                      // rebuilt when the catalogue changes
Dictionary<int, ProductDetails> cold;  // rarely used fields

6. GC and locality

• The GC's compaction tends to keep objects allocated together close together, which helps locality for freshly built structures.
• Long-lived structures that are mutated over time drift apart as new objects are allocated elsewhere.
• Rebuilding a read-mostly structure periodically (for example on catalogue refresh) restores locality; FrozenDictionary and FrozenSet do this by design.

7. Try it yourself

1. Build a list of 1 million Order objects each referencing a Customer object, shuffle the list, and sum a customer field. Then build a struct array with the same data and compare.
2. On Linux, run both under perf stat -e cycles,instructions,LLC-load-misses and compare IPC.
3. Use perf record with DOTNET_PerfMapEnabled=1 and find which method the misses are attributed to.

namespace CacheMisses;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

public sealed class Customer { public required string Name { get; init; } public int Tier { get; init; } }
public sealed class Order { public required Customer Customer { get; init; } public decimal Total { get; init; } }
public readonly record struct OrderRow(int CustomerTier, decimal Total);

[MemoryDiagnoser]
// On Windows (as administrator) add:
// [HardwareCounters(HardwareCounter.CacheMisses, HardwareCounter.LlcMisses)]
public class PointerChasing
{
    private Order[] _orders = [];
    private OrderRow[] _rows = [];

    [GlobalSetup]
    public void Setup()
    {
        var random = new Random(3);
        var customers = Enumerable.Range(0, 200_000)
            .Select(i => new Customer { Name = $"C{i}", Tier = i % 4 })
            .ToArray();
        _orders = Enumerable.Range(0, 2_000_000)
            .Select(i => new Order { Customer = customers[random.Next(customers.Length)], Total = i % 100 })
            .ToArray();
        random.Shuffle(_orders);
        _rows = _orders.Select(o => new OrderRow(o.Customer.Tier, o.Total)).ToArray();
    }

    [Benchmark(Baseline = true)]
    public decimal ObjectGraph()
    {
        decimal sum = 0;
        foreach (var order in _orders)
        {
            if (order.Customer.Tier == 3) sum += order.Total;   // two dependent loads, likely misses
        }
        return sum;
    }

    [Benchmark]
    public decimal FlatRows()
    {
        decimal sum = 0;
        foreach (ref readonly var row in _rows.AsSpan())
        {
            if (row.CustomerTier == 3) sum += row.Total;       // sequential stream
        }
        return sum;
    }
}

public static class Program
{
    public static void Main() => BenchmarkRunner.Run<PointerChasing>();
}

# Linux: compare instructions per cycle and last-level misses for a running service
perf stat -e cycles,instructions,LLC-loads,LLC-load-misses -p $(pgrep -f MyApi.dll) -- sleep 30