📁 Sangam Dhakal - Portfolio v2.0 - Netscape Navigator 4.0
_
□
×

📚 Interview question

A parallel loop that updates per-thread counters gets slower as you add cores. What is false sharing, how do you detect it, and how do you fix it in .NET?

Cache coherence and why two threads writing different variables on the same cache line slow each other down, how to recognise and measure false sharing, and fixes: thread-local accumulation, padding, and avoiding shared writes.

Senior Debugging Advanced cache-coherence concurrency cpu false-sharing parallel performance

🎯 Key points

  • Caches keep lines coherent across cores (MESI-style protocols); a write needs the line exclusively, so it invalidates other cores' copies of the whole line
  • False sharing: independent variables written by different threads share one cache line, so the line bounces between cores (40-100+ cycles per transfer, more across sockets)
  • Symptoms: high CPU, flat or negative scaling as threads are added, and no lock contention, GC pauses or thread-pool starvation to explain it
  • Detect with a scaling test (MaxDegreeOfParallelism 1, 2, 4, 8...), then confirm with hardware tools: perf c2c on Linux, Intel VTune or AMD uProf (Linux and Windows), BenchmarkDotNet hardware counters on Windows
  • Fix the design first: accumulate in locals and combine once (Parallel.For/ForEach localInit/localFinally, PLINQ Aggregate)
  • Where per-thread slots must be shared, pad each to its own line; line size depends on the CPU (64 bytes on most x64 and Arm64 servers, 128 on Apple M-series, and adjacent-line prefetch on x64), so 128-byte separation is the portable choice
  • Choosing a slot by ManagedThreadId or a hash is a separate correctness bug (lost updates), not just a performance one
  • True sharing (one hot Interlocked counter) costs the same and padding cannot help; shard the counter or reduce shared writes

⏱ 30-second answer

Cores keep their caches coherent: when one core writes to a cache line, copies of that whole 64-byte line in other cores are invalidated. If thread A updates counters[0] and thread B updates counters[1], they're logically independent but sit on the same line, so the line ping-pongs between cores on every write. That's false sharing: more threads, more invalidations, and the loop gets slower. Symptoms are high CPU with poor scaling and no locks in sight. The best fix is to accumulate in a local variable per thread and combine once at the end (Parallel.For with localInit/localFinally). Where per-thread state must live in shared memory, pad each slot to its own cache line (128 bytes is safe across x64 and Apple Arm64).

🗣 Full answer (senior)

Why it happens

  • Each core has private L1 and L2 caches. A coherence protocol (MESI and variants) ensures that only one core can hold a line in a modified state at a time.
  • Writing a line that another core holds forces an invalidation and a transfer of the line, which costs tens to over a hundred cycles, and much more across CPU sockets.
  • The protocol works on lines, not variables. Two unrelated longs 8 bytes apart behave like one shared variable.

Typical culprit

var counts = new long[Environment.ProcessorCount];
Parallel.For(0, items.Length, i =>
{
    int slot = Environment.CurrentManagedThreadId % counts.Length;
    counts[slot]++;          // neighbouring slots share a cache line
});

(This code is also incorrect: two threads can map to the same slot and lose increments. False sharing often comes with a race.)

Detecting it

  1. Scaling test: run with 1, 2, 4, 8 threads (ParallelOptions.MaxDegreeOfParallelism). Flat or falling throughput with rising CPU is the tell.
  2. No obvious contention: no locks in the profile, no Monitor waits, no thread pool starvation.
  3. Hardware evidence: on Linux, perf c2c record/perf c2c report shows cache lines with heavy cross-core "HITM" traffic and the offsets being written. Intel VTune's memory access analysis shows the same. On Windows, BenchmarkDotNet hardware counters show elevated cache misses.
  4. Code review: look for arrays or adjacent fields written by different threads: per-thread counters, statistics structs, ring buffer head and tail indexes.

Fixing it

Fix How Trade-off
Thread-local accumulation Parallel.For with localInit/localFinally, or PLINQ Aggregate, combining once Best: no shared writes at all in the loop
Padding Give each slot its own line: [StructLayout(LayoutKind.Explicit, Size = 128)] struct, or index i * 16 in a long[] Uses more memory; still shares if threads move slots
Separate objects Allocate one object per worker The GC may still place them adjacently; padding is more reliable
Reduce sharing Batch updates, use per-core sharded counters and sum on read More complex reads

.NET's own concurrency types do this: ConcurrentQueue<T> pads its head and tail indexes so producers and consumers don't false-share.

True sharing

One Interlocked.Increment counter hit by every thread is true sharing: the line must move on every increment, and padding can't help. Use sharded counters or local accumulation. For metrics, System.Diagnostics.Metrics counters and OpenTelemetry already aggregate efficiently.

Production example

A batch scoring job used a long[] of per-thread statistics updated millions of times. Moving from 4 to 16 cores made it 20% slower. perf c2c pointed at that array. Switching to local accumulation per partition with localFinally merging made it scale almost linearly to 16 cores, and also fixed a race that had been under-counting.

🔍 What the interviewer is checking

Checks advanced concurrency and performance debugging ability.

Green flags: explains coherence and cache lines, distinguishes false from true sharing, describes a scaling test and tools such as perf c2c, prefers local accumulation, knows padding sizes and caveats, notices the race in naive per-thread arrays.

Red flags: blames the thread pool or GC without evidence; suggests more threads; has never heard of cache lines.

📚 Learn in depth

1. The ping-pong, drawn

      core 0                       core 1
  L1: [ c0 c1 c2 ... ]  <-line->  [ c0 c1 c2 ... ]   one 64-byte line holds c0..c7

  core 0 writes c0  -> invalidate core 1's copy
  core 1 writes c1  -> must fetch line from core 0, invalidate core 0's copy
  core 0 writes c0  -> must fetch line back ...

Each transfer costs roughly 40–100+ cycles on one socket and more across sockets, versus about 1 cycle for an increment in L1.

2. MESI in one table

State Meaning
Modified Only this core has it, and it's dirty
Exclusive Only this core has it, clean
Shared Several cores have clean copies
Invalid This copy is stale

Writing requires the line in Modified or Exclusive state, so any other copies are invalidated first.

3. Cache line sizes and padding choices

CPU Line size Notes
Most Intel and AMD x64 64 bytes Adjacent-line prefetching can pair lines, so 128-byte separation avoids interference
Most Arm64 servers (for example Ampere, Graviton, Cobalt) 64 bytes
Apple M-series 128 bytes

Padding to 128 bytes is a safe portable choice for a small number of hot slots.

4. Padding in C#

[StructLayout(LayoutKind.Explicit, Size = 128)]
public struct PaddedLong
{
    [FieldOffset(64)] public long Value;   // value sits in the middle of its own 128-byte block
}

An array PaddedLong[] then places each value on its own line (arrays of structs are contiguous, and the element size is 128). Note the array's start isn't guaranteed to be 128-byte aligned; putting the value at offset 64 keeps neighbours at least 64 bytes away either side.

5. Where false sharing hides in .NET code

  • Per-thread or per-partition counters in arrays.
  • Hand-written ring buffers with head and tail fields next to each other.
  • Small objects allocated together and each owned by a different thread (the GC packs them closely).
  • static fields of one class updated by different threads (statics of a class are stored together).

6. Choosing the right fix

Can each thread keep its own running value until the end?
   yes -> local accumulation (localInit/localFinally, Aggregate) -- best
   no  -> does the value need to be read while running?
            occasionally -> sharded, padded counters; sum on read
            constantly   -> rethink: one contended value can't scale

7. Try it yourself

  1. Benchmark the shared long[] version, a padded version and a local-accumulation version with MaxDegreeOfParallelism set to 1, 2, 4 and 8.
  2. On Linux, run the shared version under perf c2c record -- dotnet run -c Release and look at the top shared cache line in perf c2c report.
  3. Replace the counters with a single Interlocked.Increment and observe true sharing.

💻 Code

namespace FalseSharing;

using System.Runtime.InteropServices;

[StructLayout(LayoutKind.Explicit, Size = 128)]
public struct PaddedLong
{
    [FieldOffset(64)] public long Value;
}

public static class Counting
{
    // BAD: slots share cache lines, and threads can collide on a slot (lost updates).
    public static long SharedArray(int[] items, int workers)
    {
        var counts = new long[workers];
        Parallel.For(0, workers, new ParallelOptions { MaxDegreeOfParallelism = workers }, w =>
        {
            for (var i = w; i < items.Length; i += workers)
            {
                if (items[i] % 3 == 0) counts[w]++;      // neighbours on the same line
            }
        });
        return counts.Sum();
    }

    // Better: each worker's slot is on its own cache line.
    public static long Padded(int[] items, int workers)
    {
        var counts = new PaddedLong[workers];
        Parallel.For(0, workers, new ParallelOptions { MaxDegreeOfParallelism = workers }, w =>
        {
            for (var i = w; i < items.Length; i += workers)
            {
                if (items[i] % 3 == 0) counts[w].Value++;
            }
        });
        long total = 0;
        foreach (var c in counts) total += c.Value;
        return total;
    }

    // Best: accumulate in a local, publish once per partition.
    public static long LocalAccumulation(int[] items)
    {
        long total = 0;
        Parallel.ForEach(
            System.Collections.Concurrent.Partitioner.Create(0, items.Length),
            localInit: () => 0L,
            body: (range, _, local) =>
            {
                for (var i = range.Item1; i < range.Item2; i++)
                {
                    if (items[i] % 3 == 0) local++;
                }
                return local;
            },
            localFinally: local => Interlocked.Add(ref total, local));
        return total;
    }
}

⚠ Common mistakes

  • Mistake: assuming independent variables can't interfere. Coherence works on 64-byte lines, so adjacent fields or array elements written by different threads contend as if shared. Instead: think in cache lines for any data written by multiple threads in a hot loop.

  • Mistake: adding threads to fix a slow parallel loop. With false sharing, more threads mean more invalidations, so throughput falls further. Instead: run a scaling test first; if it doesn't scale, look for shared writes before adding cores.

  • Mistake: padding everything. Padding wastes memory and cache capacity, and does nothing for true sharing of one hot variable. Instead: prefer local accumulation; pad only the few hot per-thread slots that must be shared.

  • Mistake: indexing per-thread arrays by ManagedThreadId or a hash. Several threads can map to one slot, causing lost updates, and threads move between tasks, so the mapping isn't stable. Instead: use partition-local state from Parallel.For/ForEach, or ThreadLocal<T> with care.

  • Mistake: hammering one Interlocked counter from every thread. Atomic operations are correct, but the line still bounces between cores on each increment. Instead: use sharded counters or local accumulation and combine at the end; for telemetry, use System.Diagnostics.Metrics.

  • Mistake: diagnosing by intuition alone. False sharing looks like "the CPU is busy" and is easy to confuse with lock contention or GC. Instead: confirm with scaling curves, perf c2c, VTune or hardware counters before changing code.

🏛 Senior insight: trade-offs

False sharing is hardware leaking through abstractions: the code has no locks and looks embarrassingly parallel, yet it does not scale. Seniors recognise the pattern from the scaling curve, prove it with tools, and fix the design rather than sprinkling padding.

Approach Scales Memory cost Complexity When to use
Shared array, adjacent slots No (false sharing) Low Low Never in a hot loop
Single Interlocked counter No (true sharing) Low Low Low-rate counters only
Padded per-worker slots Yes, if slots are stable 128 bytes per slot Medium Long-lived workers that must publish progress
Local accumulation + merge Yes None Low Default for data-parallel aggregation
Sharded counters, sum on read Yes Medium Medium Live metrics read while running
System.Diagnostics.Metrics Yes (runtime handles it) Low Low Telemetry in services

Where it shows up beyond Parallel.For: hand-written ring buffers with adjacent head and tail indexes, per-request statistics objects that the GC allocated next to each other, static counters on one class updated by many request threads in an ASP.NET Core service, and custom object pools. The BCL avoids it internally (for example ConcurrentQueue<T> pads its head and tail). In a typical web service, request work is mostly independent and I/O-bound, so the effort belongs in batch jobs, high-throughput pipelines and custom concurrent data structures, proven with a scaling benchmark kept in the repo.

🔗 Follow-ups the interviewer may ask

  1. When: Candidate talks about cache lines and coherence. Why: Checks the single-threaded side of cache behaviour.
    How to answer

    CPUs load memory in cache lines of 64 bytes, so after one access the neighbouring bytes are already in cache (spatial locality), and recently used data is likely still there (temporal locality). Hardware prefetchers also spot sequential access and fetch ahead. A C# int[,] is stored row-major: one row's elements are next to each other. Walking row by row uses every byte of each line and the prefetcher keeps up; walking column by column jumps a whole row each step, wasting most of each line and missing cache, often 5–10× slower for large arrays. The same logic favours arrays of structs over lists of objects scattered on the heap, and struct-of-arrays layouts for hot numeric fields.

  2. When: Candidate mentions different cache line sizes on Apple Arm64. Why: Probes memory ordering differences that also affect multi-threaded code across architectures.
    How to answer

    x64 comes from a CISC lineage with variable-length instructions and strong memory ordering. Arm64 is a RISC load/store design with fixed 4-byte instructions, more registers, and a weaker memory model. For .NET, IL and pure managed code run unchanged because the JIT targets whichever CPU it's on. What needs checking: native dependencies must have arm64 builds, container images must be multi-arch, RIDs such as linux-arm64 must be published, hand-written SIMD must have Arm64 or cross-platform paths, and code with data races that happened to work on x64 can break on Arm64. Then benchmark, because Arm64 VMs such as Azure's Cobalt-based sizes often give better price-performance for web workloads.

🛠 Real-world problems: how you handle them

Batch scoring job slows down when moved to a 16-core VM
Situation
A nightly credit-scoring job processed about 50 million records with `Parallel.For`, recording per-thread statistics in a shared `long[]`. Moving from a 4-core to a 16-core Azure VM made it 20% slower and it missed its batch window.
Signals
CPU at 100% on all 16 cores, throughput lower than on 4 cores, no lock contention or GC pauses in dotnet-counters, and statistics totals that varied slightly between runs.
Investigation
Ran a scaling test with `MaxDegreeOfParallelism` 1 to 16 (throughput peaked at 4). Profiled with `perf c2c` on a Linux VM, which showed one cache line with heavy cross-core modified hits at the statistics array offsets. Code review also found two threads could map to the same slot, explaining the varying totals.
Resolution
Replaced the shared array with partition-local accumulators using `localInit`/`localFinally`, merging once per partition. The job scaled nearly linearly to 16 cores and finished in under half the previous time, with stable totals.
Prevention
Added a scaling benchmark to the job's performance tests, a code-review checklist item for shared writes in parallel loops, and a run-time metric for job duration with an alert against the batch window.

How to tell it (STAR)

**Situation:** At [company], our nightly scoring job got [20%] slower after we moved it to a bigger [16-core] VM and started missing its batch window. **Task:** I had to make it scale and finish on time. **Action:** A scaling test showed throughput peaking at four threads with no lock or GC contention, so I suspected false sharing. `perf c2c` confirmed one hot cache line in our per-thread statistics array, and I also found a race that lost increments. I moved to partition-local accumulators merged once at the end. **Result:** The job scaled almost linearly, finished in [under half] the time, and the statistics became exact. I added a scaling check to our performance tests.

↩ Usually asked after

More in CPU

Last updated 23 Oct 2026.

🌐 Sangam's Webring

⏪ Previous 🏠 Main Next ⏩
💬 Chat with me