A bug only appears in the Release build in production and you can't reproduce it in Debug. How do you investigate and fix it?

1. A Release-only bug is a real bug

Debug and Release run the same source, but not the same program. If behaviour differs, one of these is true:

The last two aren't strictly Debug versus Release, but they're often what "only in Release" really means. Rule them in or out early.

2. A systematic investigation

 1. Get the exact artefact that's deployed (same pipeline build, same PDBs)
        |
 2. Reproduce with it, under production-like conditions
    (same image, CPU/memory limits, GC mode, load, config shape)
        |
        +-- can't reproduce --> collect evidence from production (section 4)
        |
 3. Narrow down: change ONE dimension at a time
    Release vs Debug | trimming on/off | R2R on/off | tiered PGO on/off | env | load
        |
 4. Read the code that differs (#if, Conditional, IsDevelopment)
    and look for shared mutable state, locals that must stay alive
        |
 5. Form a hypothesis and prove it with a minimal Release reproduction
        |
 6. Fix the cause, add a Release-mode regression test, close the gap in CI

Narrowing knobs

These runtime settings change how code is compiled without rebuilding, which helps confirm that the JIT is involved. Use them only to investigate, never as the fix.

If the bug appears or disappears with one of these, you've learned that optimisation or GC timing is involved, which points at a race, a missing barrier or a lifetime problem in your code.

At the code level, [MethodImpl(MethodImplOptions.NoInlining)] on a suspect method is a quick way to test an inlining hypothesis.

3. The memory model in practice

To run fast, the compiler, the JIT and the CPU may keep values in registers and reorder reads and writes, as long as a single thread can't tell the difference. Other threads can tell. Synchronisation primitives include the memory barriers that stop this:

Why the stop flag bug happens

Source:                               Optimised JIT output (conceptually):

private bool _stopping;              reg = this._stopping      <- read ONCE
                                     loop:
while (!_stopping)                     if (reg) goto done        <- never re-read
{                                      DoWork()  (inlined, no barrier)
    DoWork();                          goto loop
}                                    done:

Thread B: _stopping = true;  -> writes memory, but thread A only checks its register

Whether the JIT hoists the read depends on what the loop body contains and what gets inlined. Add a log line, and the call may act as a barrier and the bug disappears, which is why "adding logging fixed it" is a red flag, not a fix.

Early collection

In optimised code the JIT tells the GC only about locals that are still needed. After its last use, an object referenced only by a local can be collected before the method returns. Debug code extends lifetimes to the end of the method so the debugger can show the variable.

// BUG in Release: nothing references the timer after this line
var timer = new Timer(_ => Poll(), null, 0, 1000);

// FIX: keep it in a field and dispose it on shutdown
_timer = new Timer(_ => Poll(), null, 0, 1000);

4. Evidence from production

Reading the dump

dotnet-dump analyze core_20261022
> clrstack -all                      # what every managed thread is doing
> syncblk                            # which threads hold or wait for locks
> threadpool                         # queued work and worker threads
> dumpheap -type OrderExportWorker   # find the instance
> dumpobj <address>                  # read its fields, e.g. _stopping = 1
> gcroot <address>                   # what keeps an object alive (or why nothing does)

Open the dump with the PDBs from the same build, or the stacks won't have line numbers. Treat dumps as production data: they contain whatever was in memory, including personal information and secrets, so collect them with approval, store them encrypted with restricted access, and delete them when the investigation ends.

5. Common causes and their fixes

6. Proving the fix

• Write a test that reproduces the bug in Release (dotnet test -c Release), ideally failing reliably before the fix. For races, run the scenario many times or under stress.
• Tools for systematic concurrency testing, such as Microsoft Coyote, can explore thread interleavings that ordinary tests rarely hit.
• For shutdown bugs, an integration test that starts the published app, sends SIGTERM and asserts a clean stop within a time limit catches regressions.

7. Preventing the next one

• Build, test and load-test the Release artefact you deploy; never test Debug and ship Release.
• Keep PDBs for every Release build so production dumps are readable.
• Turn on threading analysers (for example Microsoft.VisualStudio.Threading.Analyzers) and trim analysers (IsTrimmable, EnableTrimAnalyzer) where relevant.
• Make shared mutable state a review checklist item, and prefer immutable data and message passing (channels) over shared fields.

8. .NET 11 awareness

Each release makes the JIT smarter (more inlining, better PGO, escape analysis and stack allocation). Code that relies on undefined behaviour can break on an upgrade even without any Release-versus-Debug change. Run your Release and concurrency tests when you trial .NET 11, and keep production on .NET 10 until they pass.

9. Try it yourself

1. Run the code example with dotnet run (Debug): it stops after one second.
2. Run dotnet run -c Release: it may never stop.
3. Set DOTNET_TieredCompilation=0 and run the Debug build: does that change anything? (It shouldn't; Debug assemblies disable JIT optimisation.)
4. Change the field to private volatile bool _stopping; and rerun in Release.
5. Replace the flag with a CancellationTokenSource and rerun.
6. While the broken Release build is spinning, run dotnet-dump collect -p <pid>, then clrstack -all and dumpobj on the worker to see _stopping already set.
7. Write a test that runs the worker in a Release build and fails if it doesn't stop within five seconds.

namespace ReleaseOnlyBug;

internal sealed class Worker
{
    private bool _stopping; // BUG: plain field shared across threads
    private long _iterations;

    public void Run()
    {
        while (!_stopping)
        {
            _iterations++;
        }
        Console.WriteLine($"Stopped after {_iterations:N0} iterations");
    }

    public void Stop() => _stopping = true;
}

internal static class Program
{
    private static void Main()
    {
        var worker = new Worker();
        var thread = new Thread(worker.Run) { IsBackground = true };
        thread.Start();

        Thread.Sleep(1000);
        worker.Stop();

        Console.WriteLine(thread.Join(TimeSpan.FromSeconds(5))
            ? "Worker stopped"
            : "Worker did NOT stop: the read was hoisted in optimised code");
    }
}

The fix in a hosted service:

namespace Orders.Export;

public sealed class OrderExportWorker(ILogger<OrderExportWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await ExportBatchAsync(stoppingToken);
            await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
        }
        logger.LogInformation("Order export stopped cleanly");
    }

    private static Task ExportBatchAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

Task.Delay throws OperationCanceledException on shutdown, which BackgroundService treats as a normal stop.