Customers report names like "José" in a CSV export opened in Excel, and an imported file shows "Jos�". How do you diagnose and fix these encoding problems?

1. The decoding matrix

                 decoder
               UTF-8          Windows-1252
  bytes   +----------------+-----------------+
  UTF-8   |  correct  é    |  mojibake  é   |
  (C3 A9) |                |                 |
          +----------------+-----------------+
  1252    |  invalid  �    |  correct  é     |
  (E9)    |                |                 |
          +----------------+-----------------+

Two symptoms, two opposite causes. Recognising the pattern on sight saves hours.

2. Double encoding, step by step

"é" --UTF-8--> C3 A9 --read as 1252--> "é" --saved as UTF-8--> C3 83 C2 A9

Each round trip makes the text longer and stranger. Seeing ÃÂ patterns means the data went through the wrong decoder at least twice.

3. Tools for looking at bytes

4. Where .NET picks encodings

5. Databases

• SQL Server VARCHAR with a non-UTF-8 collation stores single-byte code page data; Unicode characters outside that code page become ? on insert. Use NVARCHAR (or a _UTF8 collation on SQL Server 2019+).
• Parameters: EF Core and SqlClient send string as NVARCHAR by default; mapping a property to varchar (IsUnicode(false)) is a deliberate choice that loses characters outside the code page.

6. A repeatable investigation checklist

1. Reproduce with one record and capture the raw bytes at each hop.
2. Identify the pattern (mojibake, replacement, double encoding, lossy ?).
3. Find the first hop where bytes and expected encoding disagree.
4. Fix that hop by making the encoding explicit.
5. Assess damaged data: reversible or not, how many rows, which time range.
6. Add a regression test with names like José, Nguyễn, Māori, 王芳 and an emoji.

7. Try it yourself

1. Write José to a CSV with File.WriteAllText and open it in Excel by double-clicking. Then write it with new UTF8Encoding(true) and open it again.
2. Save a file as Windows-1252 from VS Code, read it with File.ReadAllText, and print the hex of the result. Then read it correctly with Encoding.GetEncoding(1252).
3. Simulate double encoding in code and write a function that detects and reverses it, with tests proving it leaves correct text alone.

namespace EncodingFixes;

using System.Text;
using System.Text.Unicode;

public static class CsvEncoding
{
    // Excel-friendly export: UTF-8 with a byte order mark.
    public static async Task WriteForExcelAsync(string path, IEnumerable<string[]> rows, CancellationToken ct)
    {
        await using var writer = new StreamWriter(path, append: false, new UTF8Encoding(encoderShouldEmitUTF8Identifier: true));
        foreach (var row in rows)
        {
            await writer.WriteLineAsync(string.Join(',', row.Select(Escape)).AsMemory(), ct);
        }
    }

    // Strict import: reject anything that isn't valid UTF-8 instead of silently inserting U+FFFD.
    public static string ReadUtf8Strict(byte[] content)
    {
        if (!Utf8.IsValid(content))
        {
            throw new InvalidDataException("File is not valid UTF-8. Ask the sender to export as UTF-8.");
        }
        var strict = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true);
        ReadOnlySpan<byte> span = content;
        if (span.StartsWith(Encoding.UTF8.Preamble))
        {
            span = span[Encoding.UTF8.Preamble.Length..];
        }
        return strict.GetString(span);
    }

    // Legacy partner that still sends Windows-1252.
    public static string ReadWindows1252(byte[] content)
    {
        Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
        return Encoding.GetEncoding(1252).GetString(content);
    }

    // Reverses one layer of UTF-8-read-as-1252 mojibake. Only run on rows proven to be damaged.
    public static string RepairMojibake(string damaged)
    {
        Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
        byte[] originalBytes = Encoding.GetEncoding(1252).GetBytes(damaged);
        return Encoding.UTF8.GetString(originalBytes);
    }

    private static string Escape(string field) =>
        field.IndexOfAny([',', '"', '\n', '\r']) >= 0 ? $"\"{field.Replace("\"", "\"\"")}\"" : field;
}

# Inspect the first bytes of a file
xxd export.csv | head -n 3
# 00000000: efbb bf4e 616d 652c ...   <- EF BB BF is the UTF-8 BOM