What is the difference between sequential and random file access? Why does it matter for performance on HDDs and SSDs, and which .NET APIs (FileOptions, RandomAccess, memory-mapped files) help with each?

1. What the device does

HDD: random read = seek arm + wait for platter rotation + transfer
      ┌───────────────────────────────┐
      │  ~4 ms seek + ~4 ms rotation + 0.02 ms data   │   per random 4 KiB
      └───────────────────────────────┘
     sequential = one seek, then a continuous stream

SSD: no moving parts, but
  - reads/writes happen in pages (4–16 KiB), erases in large blocks
  - parallelism across channels needs many outstanding requests (queue depth)
  - small random WRITES cause write amplification

2. Why one FileStream is a bottleneck for random reads

FileStream has a single Position. Two threads doing Seek then Read race each other, so you need a lock, which serialises all I/O.

Thread A: Seek(1000) ──┐            ┌── Read  ← reads at 5000! ✘
Thread B:        Seek(5000) ── Read ─┘

RandomAccess.Read(handle, buffer, offset) maps to pread/ReadFile with an explicit offset. No shared position, so many threads can read in parallel, which also gives SSDs the queue depth they need.

3. Memory-mapped files

process virtual memory
┌────────────────────────┐
│ view of data.bin       │── page fault on first touch ──► OS reads 4 KiB page from disk
│ (looks like an array)  │── later touches ──► straight from page cache, no system call
└────────────────────────┘

Pros: no system call per read once cached, the OS shares pages across processes, simple code for read-mostly data. Cons: page faults are hidden I/O (unpredictable latency on a cold cache), I/O errors surface as exceptions on memory access, and very large views need 64-bit address space.

4. Designing for sequential I/O

Many systems turn random writes into sequential ones on purpose:

• Databases write a sequential write-ahead log, then update pages later.
• LSM-tree stores (RocksDB, Cassandra) append to files and merge in the background.
• Kafka is an append-only log read sequentially by consumers.

For your own files: sort work by offset, batch small reads into larger ones, and append instead of updating in place.

5. Try it yourself

1. Create a 4 GB file. Time reading it start to end with a 1 MiB buffer.
2. Time 100,000 random 4 KiB reads at random offsets with RandomAccess.Read.
3. Repeat both after dropping the page cache (sync; echo 3 | sudo tee /proc/sys/vm/drop_caches on a test Linux box) to see cold vs warm.
4. Run the random test with 1 thread and with 16 threads (Parallel.ForAsync) and compare throughput on an SSD.

// Random, parallel, offset-based reads of fixed-size records (.NET 6+)
public sealed class RecordFile : IDisposable
{
    private const int RecordSize = 256;
    private readonly SafeFileHandle _handle;

    public RecordFile(string path) =>
        _handle = File.OpenHandle(path, FileMode.Open, FileAccess.Read, FileShare.Read,
            FileOptions.RandomAccess | FileOptions.Asynchronous);

    public long Count => RandomAccess.GetLength(_handle) / RecordSize;

    public async ValueTask<byte[]> ReadAsync(long index, CancellationToken ct)
    {
        var buffer = new byte[RecordSize];
        int read = await RandomAccess.ReadAsync(_handle, buffer, index * RecordSize, ct);
        if (read != RecordSize) throw new EndOfStreamException();
        return buffer;  // thread-safe: no shared Position, no lock needed
    }

    public void Dispose() => _handle.Dispose();
}

// Sequential scan with a larger buffer and a read-ahead hint
await using var fs = new FileStream("/data/events.log", new FileStreamOptions
{
    Mode = FileMode.Open,
    Access = FileAccess.Read,
    Share = FileShare.Read,
    BufferSize = 1024 * 1024,
    Options = FileOptions.SequentialScan | FileOptions.Asynchronous
});
using var reader = new StreamReader(fs);
while (await reader.ReadLineAsync(ct) is { } line) Handle(line);

// Memory-mapped lookup in a read-mostly index
using var mmf = MemoryMappedFile.CreateFromFile("/data/index.bin", FileMode.Open, null, 0,
    MemoryMappedFileAccess.Read);
using var view = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read);
long offset = 1_234_567L * 16;
long key = view.ReadInt64(offset);
long value = view.ReadInt64(offset + 8);