What is character encoding? Explain ASCII and Unicode, and the difference between a character, a code point and a code unit.

1. Layers of text, from screen to bytes

  what the user sees    é        👍🏽
        |
  grapheme clusters     [é]      [👍🏽]
        |
  code points           U+00E9   U+1F44D U+1F3FD
        |  encoding form
  UTF-16 code units     00E9     D83D DC4D D83C DFFD
  UTF-8 bytes           C3 A9    F0 9F 91 8D F0 9F 8F BD

2. One character, two spellings: normalisation

é can be precomposed (U+00E9) or decomposed (e U+0065 + combining acute U+0301). They look identical but compare unequal with ordinal comparison.

• NFC (composed) is the usual storage form; macOS file names historically used a decomposed form.
• string.Normalize(NormalizationForm.FormC) converts. Normalise user input before comparing or using it as a key (user names, file names).

3. ASCII control characters you still meet

Line endings cause real bugs: a CRLF file read on Linux leaves \r at the end of each line, which breaks parsing and comparisons.

4. Code pages and why they still exist

• Older systems, mainframe exports and some bank files still use Windows-1252, ISO-8859-1 or EBCDIC.
• On .NET (Core and later), only UTF-8, UTF-16, UTF-32, ASCII and Latin-1 are built in. For others, call Encoding.RegisterProvider(CodePagesEncodingProvider.Instance) once, then Encoding.GetEncoding(1252).
• Latin-1 (ISO-8859-1) and Windows-1252 differ in bytes 0x80–0x9F (Windows-1252 has €, curly quotes and dashes there).

5. Comparisons and culture

• string.Equals(a, b, StringComparison.Ordinal) compares code units: fast and predictable; use it for identifiers, keys and protocol values.
• Culture-aware comparison treats some different sequences as equal and some letters differently (the Turkish dotless ı problem with ToUpper).
• Use OrdinalIgnoreCase for case-insensitive identifiers, not ToLower() comparisons.

6. Security angles

• Homoglyphs: Cyrillic а looks like Latin a; attackers register look-alike user names and domains. Normalise and restrict allowed scripts where identity matters.
• Invalid sequences: decoders replace invalid bytes with U+FFFD (�) by default; validate input at the boundary instead of silently accepting damage.
• Overlong encodings were historically used to sneak ../ past filters; modern .NET decoders reject them.

7. Try it yourself

1. Print "é".Length for the precomposed and decomposed forms ("é"), then compare them with == and after Normalize().
2. Print "👍🏽".Length, the number of runes (EnumerateRunes().Count()), and new StringInfo("👍🏽").LengthInTextElements.
3. Encode "Café" with Encoding.ASCII, Encoding.Latin1 and Encoding.UTF8, print the bytes in hex, and decode each with the wrong encoding to see what breaks.

namespace EncodingBasics;

using System.Globalization;
using System.Text;

public static class Program
{
    public static void Main()
    {
        string composed = "é";          // é as one code point
        string decomposed = "é";       // e + combining acute accent

        Console.WriteLine($"{composed.Length} vs {decomposed.Length}");   // 1 vs 2
        Console.WriteLine(composed == decomposed);                          // False
        Console.WriteLine(composed == decomposed.Normalize(NormalizationForm.FormC)); // True

        string thumbs = "\U0001F44D\U0001F3FD";  // thumbs up + medium skin tone
        Console.WriteLine($"chars (UTF-16 code units): {thumbs.Length}");              // 4
        Console.WriteLine($"runes (code points): {thumbs.EnumerateRunes().Count()}");  // 2
        Console.WriteLine($"graphemes: {new StringInfo(thumbs).LengthInTextElements}"); // 1

        foreach (var encoding in new[] { Encoding.ASCII, Encoding.Latin1, Encoding.UTF8 })
        {
            byte[] bytes = encoding.GetBytes("Café");
            Console.WriteLine($"{encoding.WebName,-12} {Convert.ToHexString(bytes)}");
        }
        // us-ascii     4361663F      (é lost, replaced by '?')
        // iso-8859-1   436166E9
        // utf-8        436166C3A9
    }
}