What is SIMD, how do CPU vector registers work, and when would you use Vector128<T>, Vector256<T> or Vector<T> in .NET?

1. Lanes and widths

Vector<T>.Count depends on the machine (for example 8 ints with AVX2, 4 with NEON), which is convenient but makes results and performance hardware-dependent. Fixed-width Vector128<T> and Vector256<T> give predictable behaviour.

2. Anatomy of a vectorised sum

data:  [ 1  2  3  4 | 5  6  7  8 | 9 10 11 12 | 13 14 ]
         vector 0     vector 1     vector 2     scalar tail

acc  = [0 0 0 0]
acc += [1 2 3 4]      -> [ 1  2  3  4]
acc += [5 6 7 8]      -> [ 6  8 10 12]
acc += [9 10 11 12]   -> [15 18 21 24]
horizontal sum        -> 78
scalar tail 13 + 14   -> 105

3. Masks instead of branches

Comparisons such as Vector128.GreaterThan(a, b) return a vector of all-ones or all-zeros lanes. Combine with & to select values, or use Vector128.ConditionalSelect(mask, a, b). To find the first match, ExtractMostSignificantBits() turns a mask into an integer bitmap that BitOperations.TrailingZeroCount can scan. This is how IndexOf is implemented.

4. Hardware realities

• AVX-512 is available on recent Intel server CPUs and AMD Zen 4 and later; .NET 8 added Vector512<T>. On some CPUs, heavy 512-bit work lowers clock speed, so wider isn't automatically faster.
• Arm64 has 128-bit NEON on all .NET-supported chips; Vector256 is emulated there.
• Alignment: unaligned loads are fine on modern CPUs; crossing cache-line boundaries costs a little. Don't complicate code for alignment without measurements.
• Memory bandwidth: if the loop is limited by reading data from RAM, SIMD can't make RAM faster. SIMD shines when data is in cache or the work per byte is significant.

5. Correctness pitfalls

• Integer lanes can overflow (lanes wrap independently); widen to long lanes or reduce periodically.
• Floating-point results differ slightly from scalar order.
• The scalar tail is where bugs hide; test lengths 0, 1, Count - 1, Count, Count + 1 and large random lengths.
• Test both paths: set DOTNET_EnableHWIntrinsic=0 to force the fallback in a test run.

6. Try it yourself

1. Implement SumVector128(ReadOnlySpan<int>) and compare it with a scalar loop and values.Sum() (LINQ) in BenchmarkDotNet at 16, 1,000 and 1,000,000 elements.
2. Implement CountGreaterThan(ReadOnlySpan<int>, int threshold) with masks.
3. Run the tests with DOTNET_EnableHWIntrinsic=0 and confirm the fallback gives identical results.
4. Replace a hand-written cosine similarity with TensorPrimitives.CosineSimilarity and benchmark.

namespace SimdBasics;

using System.Runtime.InteropServices;
using System.Runtime.Intrinsics;

public static class VectorMath
{
    public static long Sum(ReadOnlySpan<int> values)
    {
        long total = 0;
        int i = 0;

        if (Vector128.IsHardwareAccelerated && values.Length >= Vector128<int>.Count)
        {
            // Accumulate in long lanes to avoid per-lane int overflow.
            var acc = Vector128<long>.Zero;
            ref int start = ref MemoryMarshal.GetReference(values);
            int lastBlock = values.Length - Vector128<int>.Count;
            for (; i <= lastBlock; i += Vector128<int>.Count)
            {
                var v = Vector128.LoadUnsafe(ref start, (nuint)i);
                var (lower, upper) = Vector128.Widen(v);   // 4 ints -> 2 + 2 longs
                acc += lower + upper;
            }
            total = Vector128.Sum(acc);
        }

        for (; i < values.Length; i++)      // scalar tail (and fallback)
        {
            total += values[i];
        }
        return total;
    }

    public static int CountGreaterThan(ReadOnlySpan<int> values, int threshold)
    {
        int count = 0, i = 0;
        if (Vector128.IsHardwareAccelerated)
        {
            var limit = Vector128.Create(threshold);
            for (; i <= values.Length - Vector128<int>.Count; i += Vector128<int>.Count)
            {
                var mask = Vector128.GreaterThan(Vector128.Create(values.Slice(i, Vector128<int>.Count)), limit);
                count += System.Numerics.BitOperations.PopCount(mask.ExtractMostSignificantBits());
            }
        }
        for (; i < values.Length; i++)
        {
            if (values[i] > threshold) count++;
        }
        return count;
    }
}

// Prefer the library when it already does the job (NuGet: System.Numerics.Tensors)
using System.Numerics.Tensors;

float similarity = TensorPrimitives.CosineSimilarity(queryEmbedding, documentEmbedding);