The system call mechanism, the path from .NET APIs through SafeHandles and the PAL to Windows (IOCP) and Linux (epoll) interfaces, the cost of crossing into the kernel, and how buffering, batching and async I/O reduce that cost. Includes tracing system calls with strace and Process Monitor.
A system call is how a user-mode program asks the kernel to do something privileged: read a file, send data, allocate memory, create a thread. The program puts arguments in registers and executes a special instruction (syscall on x64); the CPU switches to kernel mode, the kernel validates the request, does the work and returns. In .NET, File.ReadAllBytesAsync goes through FileStream/RandomAccess and a SafeFileHandle to ReadFile on Windows or pread on Linux; Socket.SendAsync uses I/O completion ports on Windows and epoll on Linux. Each call costs hundreds of nanoseconds to microseconds, more with security mitigations, so thousands of tiny calls add up. Buffering, batching and async I/O keep the count down.
On Windows, applications call documented Win32 APIs (ReadFile, WSASend) in user-mode DLLs, which issue the actual system calls via ntdll.dll. On Linux, the C library (libc) wraps system calls such as read, write, openat, epoll_wait.
File.ReadAllBytesAsync(path)
-> FileStream / RandomAccess
-> SafeFileHandle (owns the OS handle / file descriptor)
-> Windows: ReadFile (overlapped) Linux: pread on a thread-pool thread
-> kernel: file system, page cache, disk driver
Socket.SendAsync(buffer)
-> Socket / SocketAsyncEventArgs
-> Windows: WSASend + I/O completion port Linux: send; if it would block, register with epoll
-> kernel: TCP stack, network driver
| Cost | Why |
|---|---|
| Mode switch | Saving and restoring state, entering and leaving the kernel; mitigations for speculative-execution attacks add more |
| Validation and bookkeeping | Checking handles, copying arguments, locking kernel structures |
| Cache and TLB pollution | Kernel code and data displace your hot data |
| Blocking | If data isn't ready, the thread sleeps and a context switch follows |
| Physical I/O | If the page cache misses, the disk or network adds micro- to milliseconds |
FileStream has a 4 KB buffer by default; larger buffers (64–128 KB) help sequential throughput. Avoid Flush() per line.PipeWriter, BufferedStream, or gather writes.Stream.CopyToAsync with an appropriate buffer, RandomAccess.ReadAsync with large spans.File.Exists before File.Open is two calls and a race; just open and handle the exception. Cache directory listings.HttpClient connection pooling avoids socket creation, connect and TLS handshake syscalls.A CSV import read files line by line with StreamReader.ReadLine() over a FileStream opened with bufferSize: 1 (copied from an old sample to "avoid double buffering"). strace -c showed millions of 1-byte-ish read calls. Restoring a 64 KB buffer cut import time for a 2 GB file from 9 minutes to 70 seconds.
Checks understanding of the OS interface and I/O performance.
Green flags: explains the mechanism and validation, maps .NET APIs to OS calls, knows IOCP versus epoll, gives realistic costs, proposes buffering and batching, and knows how to trace calls.
Red flags: thinks .NET talks to hardware directly; disables buffering for speed; no idea how to see what system calls an app makes.
user mode kernel mode
--------- -----------
RandomAccess.Read(handle, span, offset)
pread(fd, buf, len, off) -- syscall --> sys_pread64
fd -> struct file (per-process table)
page cache lookup
hit: copy pages to buf
miss: issue disk I/O, sleep thread,
wake on completion, copy
<-- return n --
| Windows IOCP | Linux epoll | Linux io_uring | |
|---|---|---|---|
| Model | Completion: "this operation finished" | Readiness: "this socket is readable" | Completion via shared ring buffers |
| .NET use | Sockets and files | Sockets (event-loop threads) | Not used by default by the runtime |
| Syscalls per operation | Issue + completion dequeue (batched) | epoll_wait (batched) + the actual recv/send |
Very few; submissions batched |
Knowing the readiness versus completion difference explains why, on Linux, a large number of tiny socket operations still means many recv/send calls.
| Operation | Order of magnitude |
|---|---|
Trivial syscall (getpid) |
~100 ns |
read from page cache, small |
~0.5–2 µs |
write to a socket, small |
~1–5 µs |
fsync on SSD |
~0.1–10 ms |
| Blocking read that waits on network | Network latency plus a context switch |
At 100,000 requests per second, an extra 10 µs of syscalls per request is a whole core.
# Linux: summary of calls, counts and time for a running process
strace -c -f -p <pid>
# Linux: watch file opens only
strace -f -e trace=openat -p <pid>
# Lower overhead tracing with perf
perf trace -s -p <pid>
Windows: Process Monitor (Sysinternals) shows file, registry and network activity with stacks; ETW via PerfView or WPR/WPA shows syscalls and I/O with low overhead.
strace slows the traced process significantly; use it briefly, on a canary instance, never as a permanent fixture.
| Code | System calls |
|---|---|
File.Exists + File.ReadAllText |
stat + openat + reads + close |
Directory.GetFiles on a big directory |
Many getdents64 calls |
DateTime.Now |
Usually none (vDSO on Linux), but time zone conversion may read files once |
new HttpClient() per request |
socket, connect, TLS, close per request |
Console.WriteLine in a hot loop |
A write per call when output is unbuffered (common for console) |
FileStream buffer sizes of 1, 4,096 and 131,072 bytes. Time each and count calls with strace -c.File.Exists + File.ReadAllText with a single try/catch (FileNotFoundException) and compare the traced calls.StreamWriter wrapping Console.OpenStandardOutput() with AutoFlush = false, and compare.namespace SystemCalls;
using System.Diagnostics;
public static class Program
{
public static async Task Main(string[] args)
{
string path = args.Length > 0 ? args[0] : "big.csv";
foreach (var bufferSize in new[] { 1, 4_096, 128 * 1024 })
{
var sw = Stopwatch.StartNew();
long lines = await CountLinesAsync(path, bufferSize);
Console.WriteLine($"buffer {bufferSize,7}: {lines} lines in {sw.ElapsedMilliseconds} ms");
}
}
private static async Task<long> CountLinesAsync(string path, int bufferSize)
{
var options = new FileStreamOptions
{
Mode = FileMode.Open,
Access = FileAccess.Read,
BufferSize = bufferSize, // 0 or 1 disables FileStream buffering
Options = FileOptions.SequentialScan | FileOptions.Asynchronous,
};
await using var stream = new FileStream(path, options);
using var reader = new StreamReader(stream, bufferSize: Math.Max(bufferSize, 1024));
long count = 0;
while (await reader.ReadLineAsync() is not null)
{
count++;
}
return count;
}
}
# Compare system call counts for each run
strace -c -f -e trace=read,pread64 dotnet SystemCalls.dll big.csv
Mistake: disabling buffering "for performance".
bufferSize: 0 or 1 turns every small read or write into a system call, which is dramatically slower for typical access patterns.
Instead: keep buffering on, and increase the buffer (64–128 KB) for large sequential I/O.
Mistake: flushing after every write.
Each Flush() forces a system call, and Flush(flushToDisk: true) forces a slow fsync too.
Instead: flush at logical boundaries (end of a batch or file), and use fsync only where durability truly requires it.
Mistake: checking before acting (File.Exists then Open).
It doubles the calls and introduces a race: the file can disappear between the check and the open.
Instead: open directly and handle FileNotFoundException or use FileMode.OpenOrCreate.
Mistake: assuming async file I/O is free or always truly async. On Linux, regular file reads run on thread-pool threads, so async file APIs mainly free the caller, not the system. Instead: use async for scalability, but tune buffer sizes and avoid huge numbers of small file operations.
Mistake: leaving strace attached in production.
It can slow the process by an order of magnitude and affect users.
Instead: trace briefly on a canary instance, or use lower-overhead tools such as perf trace or ETW.
System calls are where your code meets the OS, and they are often the hidden cost behind "the CPU is busy but the managed profiler shows nothing".
| Symptom | Likely syscall pattern | Fix |
|---|---|---|
| High system CPU, low disk throughput | Millions of tiny read/write |
Restore or enlarge buffers, batch writes |
| Latency spikes on writes | fsync per message |
Flush at batch boundaries; group commit |
| Socket churn, TLS CPU | socket/connect/close per request |
Reuse HttpClient via IHttpClientFactory or a static SocketsHttpHandler |
| Thread-pool growth during file I/O on Linux | Blocking pread on pool threads |
Bound concurrency, larger reads, avoid many small files |
| Slow directory scans | Many getdents64/stat calls |
Enumerate lazily (EnumerateFiles), cache listings, avoid per-file metadata calls |
Seniors know the path from .NET APIs down to IOCP or epoll, which explains why async sockets scale, why file I/O on Linux behaves differently, and why buffer sizes matter. They gather evidence (system call counts, user versus kernel CPU) before optimising, and they fix patterns rather than micro-tuning: buffer and batch, reuse connections, remove check-then-act calls, and let the kernel do the waiting.
A context switch is the OS stopping one thread and starting another on a core: it saves the registers, instruction pointer and stack pointer of the old thread and restores those of the new one. Switching between processes also swaps the address space, which is more expensive. The direct cost is a few microseconds, but the indirect cost is often bigger: the new thread finds caches and TLBs cold. Switches are voluntary when a thread blocks, and involuntary when its time slice ends. Too many threads in a .NET service, usually caused by blocking code that makes the thread pool inject more threads, means more switching, more cache misses, more memory for stacks and more lock contention, so throughput falls and latency rises. The fix is async I/O, bounded parallelism around the core count, and less lock contention.
A file descriptor (Linux, a small integer) or handle (Windows, an opaque value) is the process's reference to a kernel object: an open file, socket, pipe, event, mutex, process or thread. The kernel owns the object, and the process holds an index into its own table.
They are limited (e.g. ulimit -n often 1,024 soft by default on Linux shells, much higher in many containers and under systemd), so leaks eventually break the app.
In .NET, SafeHandle wraps the raw value: it releases it in a critical finaliser if you forget, guarantees release even under thread aborts or async exceptions, and prevents handle recycling bugs by reference counting during P/Invoke calls. You still call Dispose (using) to release deterministically.
The kernel is the core of the operating system. It schedules threads on CPUs, manages memory, talks to devices through drivers, and implements file systems, networking and security. The CPU has privilege levels: in kernel mode (ring 0 on x64, EL1 on Arm64) code can access all memory and hardware; in user mode (ring 3, EL0) it can't touch hardware or other processes' memory directly. Every .NET app runs in user mode. When it needs the kernel (to read a file, send on a socket, allocate memory), it makes a system call, the CPU switches to kernel mode, the kernel does the work and returns. Interrupts and faults also switch modes. The split protects the system: a bug in your app crashes your process, not the machine.
How to tell it (STAR)
**Situation:** At [company], our nightly CSV import of [2 GB] files took [9 minutes] each and was about to overrun the batch window. **Task:** I had to speed it up without rewriting the import. **Action:** System CPU was very high while disk throughput was low, so I traced system calls with `strace -c` and found millions of tiny reads. The stream had been created with a one-byte buffer. I set a [128 KB] buffer with sequential-scan hints and matched the reader's buffer. **Result:** The import dropped to about [70 seconds] per file, freeing the batch window, and a performance test now guards it.
Last updated 23 Oct 2026.