Cache coherence and why two threads writing different variables on the same cache line slow each other down, how to recognise and measure false sharing, and fixes: thread-local accumulation, padding, and avoiding shared writes.
Cores keep their caches coherent: when one core writes to a cache line, copies of that whole 64-byte line in other cores are invalidated. If thread A updates counters[0] and thread B updates counters[1], they're logically independent but sit on the same line, so the line ping-pongs between cores on every write. That's false sharing: more threads, more invalidations, and the loop gets slower. Symptoms are high CPU with poor scaling and no locks in sight. The best fix is to accumulate in a local variable per thread and combine once at the end (Parallel.For with localInit/localFinally). Where per-thread state must live in shared memory, pad each slot to its own cache line (128 bytes is safe across x64 and Apple Arm64).
longs 8 bytes apart behave like one shared variable.var counts = new long[Environment.ProcessorCount];
Parallel.For(0, items.Length, i =>
{
int slot = Environment.CurrentManagedThreadId % counts.Length;
counts[slot]++; // neighbouring slots share a cache line
});
(This code is also incorrect: two threads can map to the same slot and lose increments. False sharing often comes with a race.)
ParallelOptions.MaxDegreeOfParallelism). Flat or falling throughput with rising CPU is the tell.Monitor waits, no thread pool starvation.perf c2c record/perf c2c report shows cache lines with heavy cross-core "HITM" traffic and the offsets being written. Intel VTune's memory access analysis shows the same. On Windows, BenchmarkDotNet hardware counters show elevated cache misses.| Fix | How | Trade-off |
|---|---|---|
| Thread-local accumulation | Parallel.For with localInit/localFinally, or PLINQ Aggregate, combining once |
Best: no shared writes at all in the loop |
| Padding | Give each slot its own line: [StructLayout(LayoutKind.Explicit, Size = 128)] struct, or index i * 16 in a long[] |
Uses more memory; still shares if threads move slots |
| Separate objects | Allocate one object per worker | The GC may still place them adjacently; padding is more reliable |
| Reduce sharing | Batch updates, use per-core sharded counters and sum on read | More complex reads |
.NET's own concurrency types do this: ConcurrentQueue<T> pads its head and tail indexes so producers and consumers don't false-share.
One Interlocked.Increment counter hit by every thread is true sharing: the line must move on every increment, and padding can't help. Use sharded counters or local accumulation. For metrics, System.Diagnostics.Metrics counters and OpenTelemetry already aggregate efficiently.
A batch scoring job used a long[] of per-thread statistics updated millions of times. Moving from 4 to 16 cores made it 20% slower. perf c2c pointed at that array. Switching to local accumulation per partition with localFinally merging made it scale almost linearly to 16 cores, and also fixed a race that had been under-counting.
Checks advanced concurrency and performance debugging ability.
Green flags: explains coherence and cache lines, distinguishes false from true sharing, describes a scaling test and tools such as perf c2c, prefers local accumulation, knows padding sizes and caveats, notices the race in naive per-thread arrays.
Red flags: blames the thread pool or GC without evidence; suggests more threads; has never heard of cache lines.
core 0 core 1
L1: [ c0 c1 c2 ... ] <-line-> [ c0 c1 c2 ... ] one 64-byte line holds c0..c7
core 0 writes c0 -> invalidate core 1's copy
core 1 writes c1 -> must fetch line from core 0, invalidate core 0's copy
core 0 writes c0 -> must fetch line back ...
Each transfer costs roughly 40–100+ cycles on one socket and more across sockets, versus about 1 cycle for an increment in L1.
| State | Meaning |
|---|---|
| Modified | Only this core has it, and it's dirty |
| Exclusive | Only this core has it, clean |
| Shared | Several cores have clean copies |
| Invalid | This copy is stale |
Writing requires the line in Modified or Exclusive state, so any other copies are invalidated first.
| CPU | Line size | Notes |
|---|---|---|
| Most Intel and AMD x64 | 64 bytes | Adjacent-line prefetching can pair lines, so 128-byte separation avoids interference |
| Most Arm64 servers (for example Ampere, Graviton, Cobalt) | 64 bytes | |
| Apple M-series | 128 bytes |
Padding to 128 bytes is a safe portable choice for a small number of hot slots.
[StructLayout(LayoutKind.Explicit, Size = 128)]
public struct PaddedLong
{
[FieldOffset(64)] public long Value; // value sits in the middle of its own 128-byte block
}
An array PaddedLong[] then places each value on its own line (arrays of structs are contiguous, and the element size is 128). Note the array's start isn't guaranteed to be 128-byte aligned; putting the value at offset 64 keeps neighbours at least 64 bytes away either side.
head and tail fields next to each other.static fields of one class updated by different threads (statics of a class are stored together).Can each thread keep its own running value until the end?
yes -> local accumulation (localInit/localFinally, Aggregate) -- best
no -> does the value need to be read while running?
occasionally -> sharded, padded counters; sum on read
constantly -> rethink: one contended value can't scale
long[] version, a padded version and a local-accumulation version with MaxDegreeOfParallelism set to 1, 2, 4 and 8.perf c2c record -- dotnet run -c Release and look at the top shared cache line in perf c2c report.Interlocked.Increment and observe true sharing.namespace FalseSharing;
using System.Runtime.InteropServices;
[StructLayout(LayoutKind.Explicit, Size = 128)]
public struct PaddedLong
{
[FieldOffset(64)] public long Value;
}
public static class Counting
{
// BAD: slots share cache lines, and threads can collide on a slot (lost updates).
public static long SharedArray(int[] items, int workers)
{
var counts = new long[workers];
Parallel.For(0, workers, new ParallelOptions { MaxDegreeOfParallelism = workers }, w =>
{
for (var i = w; i < items.Length; i += workers)
{
if (items[i] % 3 == 0) counts[w]++; // neighbours on the same line
}
});
return counts.Sum();
}
// Better: each worker's slot is on its own cache line.
public static long Padded(int[] items, int workers)
{
var counts = new PaddedLong[workers];
Parallel.For(0, workers, new ParallelOptions { MaxDegreeOfParallelism = workers }, w =>
{
for (var i = w; i < items.Length; i += workers)
{
if (items[i] % 3 == 0) counts[w].Value++;
}
});
long total = 0;
foreach (var c in counts) total += c.Value;
return total;
}
// Best: accumulate in a local, publish once per partition.
public static long LocalAccumulation(int[] items)
{
long total = 0;
Parallel.ForEach(
System.Collections.Concurrent.Partitioner.Create(0, items.Length),
localInit: () => 0L,
body: (range, _, local) =>
{
for (var i = range.Item1; i < range.Item2; i++)
{
if (items[i] % 3 == 0) local++;
}
return local;
},
localFinally: local => Interlocked.Add(ref total, local));
return total;
}
}
Mistake: assuming independent variables can't interfere. Coherence works on 64-byte lines, so adjacent fields or array elements written by different threads contend as if shared. Instead: think in cache lines for any data written by multiple threads in a hot loop.
Mistake: adding threads to fix a slow parallel loop. With false sharing, more threads mean more invalidations, so throughput falls further. Instead: run a scaling test first; if it doesn't scale, look for shared writes before adding cores.
Mistake: padding everything. Padding wastes memory and cache capacity, and does nothing for true sharing of one hot variable. Instead: prefer local accumulation; pad only the few hot per-thread slots that must be shared.
Mistake: indexing per-thread arrays by ManagedThreadId or a hash.
Several threads can map to one slot, causing lost updates, and threads move between tasks, so the mapping isn't stable.
Instead: use partition-local state from Parallel.For/ForEach, or ThreadLocal<T> with care.
Mistake: hammering one Interlocked counter from every thread.
Atomic operations are correct, but the line still bounces between cores on each increment.
Instead: use sharded counters or local accumulation and combine at the end; for telemetry, use System.Diagnostics.Metrics.
Mistake: diagnosing by intuition alone.
False sharing looks like "the CPU is busy" and is easy to confuse with lock contention or GC.
Instead: confirm with scaling curves, perf c2c, VTune or hardware counters before changing code.
False sharing is hardware leaking through abstractions: the code has no locks and looks embarrassingly parallel, yet it does not scale. Seniors recognise the pattern from the scaling curve, prove it with tools, and fix the design rather than sprinkling padding.
| Approach | Scales | Memory cost | Complexity | When to use |
|---|---|---|---|---|
| Shared array, adjacent slots | No (false sharing) | Low | Low | Never in a hot loop |
Single Interlocked counter |
No (true sharing) | Low | Low | Low-rate counters only |
| Padded per-worker slots | Yes, if slots are stable | 128 bytes per slot | Medium | Long-lived workers that must publish progress |
| Local accumulation + merge | Yes | None | Low | Default for data-parallel aggregation |
| Sharded counters, sum on read | Yes | Medium | Medium | Live metrics read while running |
System.Diagnostics.Metrics |
Yes (runtime handles it) | Low | Low | Telemetry in services |
Where it shows up beyond Parallel.For: hand-written ring buffers with adjacent head and tail indexes, per-request statistics objects that the GC allocated next to each other, static counters on one class updated by many request threads in an ASP.NET Core service, and custom object pools. The BCL avoids it internally (for example ConcurrentQueue<T> pads its head and tail). In a typical web service, request work is mostly independent and I/O-bound, so the effort belongs in batch jobs, high-throughput pipelines and custom concurrent data structures, proven with a scaling benchmark kept in the repo.
CPUs load memory in cache lines of 64 bytes, so after one access the neighbouring bytes are already in cache (spatial locality), and recently used data is likely still there (temporal locality). Hardware prefetchers also spot sequential access and fetch ahead. A C# int[,] is stored row-major: one row's elements are next to each other. Walking row by row uses every byte of each line and the prefetcher keeps up; walking column by column jumps a whole row each step, wasting most of each line and missing cache, often 5–10× slower for large arrays. The same logic favours arrays of structs over lists of objects scattered on the heap, and struct-of-arrays layouts for hot numeric fields.
x64 comes from a CISC lineage with variable-length instructions and strong memory ordering. Arm64 is a RISC load/store design with fixed 4-byte instructions, more registers, and a weaker memory model. For .NET, IL and pure managed code run unchanged because the JIT targets whichever CPU it's on. What needs checking: native dependencies must have arm64 builds, container images must be multi-arch, RIDs such as linux-arm64 must be published, hand-written SIMD must have Arm64 or cross-platform paths, and code with data races that happened to work on x64 can break on Arm64. Then benchmark, because Arm64 VMs such as Azure's Cobalt-based sizes often give better price-performance for web workloads.
How to tell it (STAR)
**Situation:** At [company], our nightly scoring job got [20%] slower after we moved it to a bigger [16-core] VM and started missing its batch window. **Task:** I had to make it scale and finish on time. **Action:** A scaling test showed throughput peaking at four threads with no lock or GC contention, so I suspected false sharing. `perf c2c` confirmed one hot cache line in our per-thread statistics array, and I also found a race that lost increments. I moved to partition-local accumulators merged once at the end. **Result:** The job scaled almost linearly, finished in [under half] the time, and the statistics became exact. I added a scaling check to our performance tests.
Last updated 23 Oct 2026.