Explain the layers of buffering between your .NET code and the disk. What does Flush actually guarantee, and how do you write a file so it survives a crash or power loss?

1. FileStream buffering

• Default buffer size is 4,096 bytes. Pass bufferSize: 0 or 1 to disable it (useful when you already write large blocks).
• Larger buffers (64–128 KiB) help sequential throughput for big files; beyond that the gain is small.
• StreamWriter has its own char buffer and an AutoFlush property. AutoFlush = true flushes on every Write, which is safe but slow.
• StreamWriter does not flush in a finaliser. If you never dispose it, the last buffer is simply lost.

2. The OS page cache

The kernel keeps file data in RAM after reads and writes:

• Reads: the second read of a file comes from RAM. That is why benchmarks must account for cold vs warm cache.
• Writes: data becomes "dirty" pages that the kernel writes back later (on Linux usually within ~30 seconds, controlled by vm.dirty_* settings).
• free -m on Linux shows this as buff/cache; it is reclaimable and not a memory leak.

3. Atomic replace

1. write  orders.json.tmp   (complete content)
2. fsync  orders.json.tmp   (Flush(true))
3. rename orders.json.tmp → orders.json   (atomic on the same file system)
4. (Linux, strict) fsync the directory so the rename itself is durable

Reader sees:  old orders.json  ───────────►  new orders.json
              never a half-written file

Rename is atomic within one volume on POSIX and NTFS. Across volumes it becomes copy plus delete, which is not atomic, so keep the temp file in the same folder. .NET does not expose directory fsync directly; databases and some libraries do it via native calls when they need the strictest guarantee.

4. Buffer management in code

• Rent large temporary buffers from ArrayPool<byte>.Shared instead of allocating new byte[1_000_000] per request (avoids LOH churn).
• Use stackalloc or Span<byte> for small, short-lived buffers.
• Always Return rented arrays in finally, and do not use them after returning.

5. Cost of durability

fsync latency on a cloud SSD can be 1–10 ms. Calling it per record limits you to hundreds of writes per second per file. Databases solve this with group commit: batch many transactions into one fsync. Apply the same idea: batch writes, then sync once.

6. Try it yourself

1. Write 1 million short lines with bufferSize: 1 and AutoFlush = true, then with a 64 KiB buffer. Compare times.
2. Write a file without disposing the StreamWriter and exit the process: see the missing tail.
3. On Linux, run strace -f -e trace=write,fsync dotnet App.dll to see how many write and fsync calls each version makes.

// Durable, atomic file replace with safe cleanup
public static async Task WriteAtomicallyAsync(
    string path, ReadOnlyMemory<byte> content, ILogger logger, CancellationToken ct)
{
    var directory = Path.GetDirectoryName(Path.GetFullPath(path))!;
    var tempPath = Path.Join(directory, $".{Path.GetFileName(path)}.{Guid.NewGuid():N}.tmp");

    try
    {
        await using (var fs = new FileStream(tempPath, new FileStreamOptions
        {
            Mode = FileMode.CreateNew,
            Access = FileAccess.Write,
            Options = FileOptions.Asynchronous,
            BufferSize = 0 // one large write; skip the extra copy
        }))
        {
            await fs.WriteAsync(content, ct);
            fs.Flush(flushToDisk: true);   // fsync / FlushFileBuffers
        }

        File.Move(tempPath, path, overwrite: true); // atomic on the same volume
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Atomic write to {Path} failed", path);
        try
        {
            File.Delete(tempPath);
        }
        catch (Exception cleanupEx)
        {
            // Do not hide the original failure; a startup sweep removes stray .tmp files
            logger.LogWarning(cleanupEx, "Could not delete temp file {TempPath}", tempPath);
        }
        throw;
    }
}

// High-volume sequential write: larger buffer, one fsync for the whole file
await using var fs = new FileStream("export.csv", FileMode.Create, FileAccess.Write,
    FileShare.Read, bufferSize: 64 * 1024, useAsync: true);
await using var writer = new StreamWriter(fs, new UTF8Encoding(false)); // AutoFlush stays false

await foreach (var row in rows.WithCancellation(ct))
    await writer.WriteLineAsync(row.ToCsv());

await writer.FlushAsync(ct);   // .NET buffers -> OS
fs.Flush(flushToDisk: true);   // OS -> disk, once

// Pooled buffer for copying between streams without allocating per call
byte[] buffer = ArrayPool<byte>.Shared.Rent(81_920);
try
{
    int n;
    while ((n = await source.ReadAsync(buffer, ct)) > 0)
        await destination.WriteAsync(buffer.AsMemory(0, n), ct);
}
finally
{
    ArrayPool<byte>.Shared.Return(buffer); // never touch buffer after this
}