How do UTF-8 and UTF-16 differ, and why can string.Length give the wrong answer for user-visible characters in .NET?

1. UTF-8 bit layout

code point range        bytes   bit pattern
U+0000  - U+007F        1       0xxxxxxx
U+0080  - U+07FF        2       110xxxxx 10xxxxxx
U+0800  - U+FFFF        3       1110xxxx 10xxxxxx 10xxxxxx
U+10000 - U+10FFFF      4       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Worked example, é = U+00E9 = 000 1110 1001:

110 00011   10 101001
  C    3     A    9     -> C3 A9

Properties that make UTF-8 robust: lead bytes and continuation bytes (10xxxxxx) never overlap, so a decoder can resynchronise after an error, and plain ASCII bytes never appear inside a multi-byte sequence (so searching for / or " in UTF-8 bytes is safe).

2. UTF-16 surrogate maths

code point  U+1F600 (😀)
subtract    0x10000          -> 0x0F600 (20 bits)
high 10 bits 0x003D + 0xD800 -> 0xD83D
low 10 bits  0x0200 + 0xDC00 -> 0xDE00
UTF-16: D83D DE00

3. Byte order marks

Encoding.UTF8 emits a BOM when used with StreamWriter or File.WriteAllText(path, text, Encoding.UTF8). new UTF8Encoding(false) and the default for File.WriteAllText(path, text) don't. StreamReader detects a BOM by default.

4. Three lengths, three limits

SQL Server VARCHAR(n) with a UTF-8 collation (SQL Server 2019+) counts bytes, so VARCHAR(50) might hold only 12 CJK characters.

5. Why .NET is UTF-16 inside and UTF-8 outside

• .NET (like Windows NT, Java and JavaScript) chose UTF-16 when Unicode was believed to fit in 16 bits. Changing it would break char and every string API.
• Instead, modern .NET does UTF-8 work at the edges without allocating strings: Utf8JsonReader and Utf8JsonWriter, Kestrel's request parsing, u8 literals, IUtf8SpanFormattable (.NET 8+). This avoids transcoding costs on hot paths.

6. Performance notes

• UTF-16 to UTF-8 transcoding is vectorised in .NET and very fast, but allocation still costs; prefer Encoding.UTF8.GetBytes(string, Span<byte>) with a pooled buffer in hot code.
• For mostly-ASCII data, UTF-8 halves memory and bandwidth compared with UTF-16. That's why JSON and HTTP use it.

7. Try it yourself

1. Print the UTF-8 and UTF-16 bytes (Encoding.Unicode) for A, é, 汉 and 😀 and explain each length.
2. Take "Hello 👋🏽 world", cut it with Substring(0, 7), encode it to UTF-8, and look for EF BF BD (the replacement character).
3. Write TruncateGraphemes(string text, int max) using StringInfo.GetNextTextElementLength and test it with the family emoji.
4. Create a SQL Server table with NVARCHAR(5) and insert five emoji. Observe the truncation error, and explain it in code units.

namespace Utf8Utf16;

using System.Globalization;
using System.Text;
using System.Text.Unicode;

public static class TextLimits
{
    /// Truncates to at most maxGraphemes user-visible characters, never splitting one.
    public static string TruncateGraphemes(string text, int maxGraphemes)
    {
        ReadOnlySpan<char> span = text;
        int index = 0, count = 0;
        while (index < span.Length && count < maxGraphemes)
        {
            index += StringInfo.GetNextTextElementLength(span[index..]);
            count++;
        }
        return text[..index];
    }

    /// Truncates so the UTF-8 encoding fits in maxBytes, without splitting a grapheme.
    public static string TruncateUtf8Bytes(string text, int maxBytes)
    {
        ReadOnlySpan<char> span = text;
        int index = 0, bytes = 0;
        while (index < span.Length)
        {
            int length = StringInfo.GetNextTextElementLength(span[index..]);
            int elementBytes = Encoding.UTF8.GetByteCount(span.Slice(index, length));
            if (bytes + elementBytes > maxBytes)
            {
                break;
            }
            bytes += elementBytes;
            index += length;
        }
        return text[..index];
    }
}

public static class Program
{
    public static void Main()
    {
        string family = "\U0001F468‍\U0001F469‍\U0001F467";
        Console.WriteLine(family.Length);                                  // 8 UTF-16 code units
        Console.WriteLine(family.EnumerateRunes().Count());                // 5 code points
        Console.WriteLine(new StringInfo(family).LengthInTextElements);    // 1 grapheme
        Console.WriteLine(Encoding.UTF8.GetByteCount(family));             // 18 bytes

        string text = "Kia ora 👋🏽 team";
        Console.WriteLine(TextLimits.TruncateGraphemes(text, 9));          // "Kia ora 👋🏽"
        Console.WriteLine(TextLimits.TruncateUtf8Bytes(text, 10));         // "Kia ora "

        ReadOnlySpan<byte> utf8 = "Café"u8;                               // UTF-8 literal (C# 11)
        Console.WriteLine(Utf8.IsValid(utf8));                            // True (.NET 8+)
    }
}