How do you work with text encodings in C#? Explain the Encoding class, UTF-8 with and without a BOM, legacy code pages, u8 literals, Rune, grapheme clusters and Unicode normalisation.

1. How UTF-8 encodes code points

Properties that matter:

• ASCII is valid UTF-8 — why UTF-8 won the web.
• Self-synchronising: continuation bytes always start with 10, so you can find character boundaries from any position.
• No byte-order issues, hence no BOM needed.
• Invalid sequences exist (overlong forms, lone continuation bytes, encoded surrogates). .NET's decoders replace invalid bytes with U+FFFD (�) by default; use new UTF8Encoding(false, throwOnInvalidBytes: true) when you must reject corrupt input.

2. UTF-16 and surrogates

.NET strings are sequences of 16-bit code units. Code points above U+FFFF are split into a high surrogate (U+D800–U+DBFF) and a low surrogate (U+DC00–U+DFFF). A lone surrogate is invalid Unicode but can exist in a .NET string (strings are not validated). Encoding such a string to UTF-8 replaces the lone surrogate with U+FFFD, silently changing data. char.IsSurrogate, char.IsSurrogatePair and Rune.TryCreate help validate.

3. Encoders, decoders and streaming

Encoding.GetString(bytes) assumes the bytes contain whole characters. When reading a stream in chunks, a multi-byte character can be split across chunk boundaries. Decoder (from encoding.GetDecoder()) keeps state between calls, which is what StreamReader uses internally. Writing your own chunked reader with GetString per buffer corrupts characters at boundaries — a classic bug in socket code.

System.IO.Pipelines + Utf8JsonReader or SequenceReader<byte> handle UTF-8 across segment boundaries without decoding to UTF-16.

4. Performance of transcoding

Modern .NET transcodes UTF-8 ↔ UTF-16 with SIMD (AVX2/AVX-512, Arm64 AdvSimd), often at several GB/s for ASCII-heavy text. Still, avoiding transcoding entirely is faster:

• Kestrel parses HTTP headers as UTF-8/ASCII bytes.
• System.Text.Json reads and writes UTF-8 directly.
• u8 literals avoid runtime encoding of constants.
• The Ascii class (.NET 8) offers fast ASCII-only validation and case-insensitive comparison on byte spans.

5. Rune and text-element APIs

string s = "Ngô 😀 é";
Console.WriteLine(s.Length);                               // UTF-16 code units
Console.WriteLine(s.EnumerateRunes().Count());             // code points
Console.WriteLine(new StringInfo(s).LengthInTextElements); // graphemes

Safe truncation by user-perceived characters:

static string TruncateGraphemes(string s, int max)
{
    var e = StringInfo.GetTextElementEnumerator(s);
    int count = 0, end = 0;
    while (count < max && e.MoveNext()) { end = e.ElementIndex + ((string)e.Current).Length; count++; }
    return s[..end];
}

(.NET 5+ also offers StringInfo.GetNextTextElementLength(ReadOnlySpan<char>) for allocation-free iteration over grapheme clusters.)

For database limits measured in bytes (e.g. a varchar UTF-8 column with a byte limit), truncate by encoded byte count while respecting boundaries.

6. Normalisation forms

string.IsNormalized() is cheap for ASCII and avoids allocation when already normalised. Note: normalisation of non-ASCII text depends on ICU data, so in globalisation-invariant mode it is limited — test it in your production container image.

7. Databases

SQL Server nvarchar stores UTF-16; varchar with a _UTF8 collation (SQL Server 2019+) stores UTF-8. Length limits are in byte-pairs for nvarchar(n) and bytes for UTF-8 varchar(n), not characters — emoji take 2 units in nvarchar and 4 bytes in UTF-8. EF Core's HasMaxLength maps to these units, so validation by string.Length can still overflow a UTF-8 varchar column.

Try it yourself

1. Write "Café, Ngô" to a CSV with and without BOM and open both in Excel.
2. Print Length, rune count and text-element count for strings with emoji, flags and combining accents.
3. Decode a UTF-8 byte array split in the middle of é using GetString per chunk vs a Decoder.
4. Compare "é" == "é" before and after Normalize().

using System.Globalization;
using System.Text;

namespace Strings;

public static class Encodings
{
    private static readonly UTF8Encoding Utf8NoBom = new(encoderShouldEmitUTF8Identifier: false);
    private static readonly UTF8Encoding Utf8Strict = new(false, throwOnInvalidBytes: true);

    public static void Main()
    {
        // 1. BOM or not
        Console.WriteLine(Encoding.UTF8.GetPreamble().Length);   // 3 (EF BB BF)
        Console.WriteLine(Utf8NoBom.GetPreamble().Length);       // 0

        // 2. Strict decoding
        byte[] latin1Bytes = [0x4A, 0x6F, 0x73, 0xE9];           // "José" in Windows-1252
        Console.WriteLine(Encoding.UTF8.GetString(latin1Bytes)); // Jos� (silent replacement)
        try { Utf8Strict.GetString(latin1Bytes); }
        catch (DecoderFallbackException) { Console.WriteLine("Invalid UTF-8 detected"); }

        // 3. Legacy code page
        Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
        var win1252 = Encoding.GetEncoding(1252);
        Console.WriteLine(win1252.GetString(latin1Bytes));      // José

        // 4. UTF-8 literal: bytes at compile time
        ReadOnlySpan<byte> contentType = "application/json"u8;
        Console.WriteLine(contentType.Length);                  // 16

        // 5. Characters users see
        string thumbs = "\U0001F44D\U0001F3FD";                   // thumbs up, medium skin tone
        Console.WriteLine(thumbs.Length);                        // 4 UTF-16 code units
        Console.WriteLine(thumbs.EnumerateRunes().Count());      // 2 code points
        Console.WriteLine(new StringInfo(thumbs).LengthInTextElements); // 1 grapheme

        // 6. Normalise before comparing
        string composed = "café", decomposed = "café";
        Console.WriteLine(composed == decomposed);                                    // False
        Console.WriteLine(composed.Normalize() == decomposed.Normalize());            // True (NFC)
    }

    // CSV for Excel users: deliberately write a BOM
    public static void WriteExcelCsv(string path, IEnumerable<string> lines)
        => File.WriteAllLines(path, lines, new UTF8Encoding(encoderShouldEmitUTF8Identifier: true));
}