📁 Sangam Dhakal - Portfolio v2.0 - Netscape Navigator 4.0
_
□
×

📚 Interview question

Your team's builds have gone from 2 minutes to 15 and incremental builds rebuild everything. How do you find out why, using MSBuild binary logs?

Diagnosing slow and non-incremental builds: capturing binary logs with -bl, reading them in the MSBuild Structured Log Viewer, finding slow targets, tasks and analysers, spotting broken incrementality, and fixes such as Inputs/Outputs, caching and build server settings.

Senior Debugging Advanced binary-log build-performance debugging incremental-build msbuild

🎯 Key points

  • Capture evidence first: dotnet build -bl writes msbuild.binlog with every property, item, target, task and timing
  • Open it in the MSBuild Structured Log Viewer and use the timeline and per-target and per-task summaries to find the slow parts
  • For incrementality, build twice and check why targets ran the second time; CoreCompile running again means an input changed
  • Common causes: custom targets without Inputs/Outputs, files written on every build, CopyToOutputDirectory=Always, globbing huge folders, slow analysers
  • Use -clp:PerformanceSummary for a quick text summary; -v:diag for detail when needed
  • Analyser cost can be measured with -p:ReportAnalyzer=true, which reports time per analyser
  • Binary logs can contain secrets and environment variables, so treat them as sensitive artefacts
  • Fix the cause, then guard it: a CI job that times a no-change rebuild

⏱ 30-second answer

First I get evidence instead of guessing. dotnet build -bl writes a binary log with every property, item, target, task and timing, and I open it in the MSBuild Structured Log Viewer. For the slow build I look at the timeline and the target and task summaries. That shows whether time goes on restore, compilation, analysers or a custom target. For the broken incremental build, I build twice and look at why targets ran the second time. If CoreCompile runs again, one of its inputs changed. Usually a custom target rewrites a generated file on every build, or something uses CopyToOutputDirectory="Always". Other culprits are globs over huge folders and slow analysers, which -p:ReportAnalyzer=true measures. I fix the cause with Inputs/Outputs, by writing files only when their content changes, and by narrowing globs. Then I add a CI check that times a no-change rebuild.

🗣 Full answer (senior)

How I would answer this in an interview

"My approach is to get evidence before guessing: dotnet build -bl writes a binary log with every property, item, target, task and timing, and I open it in the MSBuild Structured Log Viewer, using the timeline and the target and task summaries for the slow build, with -clp:PerformanceSummary as a quick text view and -v:diag only when I need more. For incrementality I build twice and read why targets ran the second time: if CoreCompile reruns, an input changed, usually a custom target without Inputs and Outputs, a file rewritten every build, CopyToOutputDirectory="Always", a glob over a huge folder, or a slow analyser, which -p:ReportAnalyzer=true measures. The mistake I see is guessing or buying bigger agents, which leaves the real cause in place.

For example, our [70-project solution at company] had slowed from [2] to [15] minutes, and even no-change builds recompiled everything. My task was to find the cause, since it was costing every developer [hours] a week. I compared binary logs of two consecutive builds and saw CoreCompile rerunning because a shared target in Directory.Build.targets wrote a timestamp into generated code every time. I made it write stable content locally and only when it changed, moved one expensive analyser to a nightly job, and added a CI guard on no-change build time. As a result, full builds dropped to [about 3] minutes and no-change builds to [about 10] seconds.

Build performance decays quietly, one target or package at a time, so I treat it like production performance: measure, fix the mechanism, then guard it with a metric in CI. And I handle binary logs as sensitive artefacts, because they capture environment variables and properties that can include secrets."


Step 1: Capture

dotnet build -bl:full.binlog                 # clean or normal build
dotnet build -bl:incremental.binlog          # immediately again, with no changes
dotnet build -clp:PerformanceSummary         # quick text summary of target and task times

For analyser cost: dotnet build -bl -p:ReportAnalyzer=true adds per-analyser timing to the log.

Step 2: Read

Open the logs in the MSBuild Structured Log Viewer (Windows app, or the cross-platform version / web viewer):

  • Timeline: which projects and targets ran in parallel and where the critical path is.
  • Target and task summaries: total time per target (CoreCompile, ResolvePackageAssets, custom targets) and per task (Csc, Copy, Exec).
  • Search: find a property ($property TreatWarningsAsErrors) or a target to see where it was set or why it ran.

Step 3: Find broken incrementality

In the second (no-change) build, every target should be skipped or trivially fast. For each target that ran, the log says why, for example "Input file X is newer than output file Y". Usual causes:

Cause Symptom Fix
Custom target without Inputs/Outputs Runs every build Declare them
A target writes a generated .cs file every time CoreCompile reruns in every project Write only if content changed (WriteLinesToFile with WriteOnlyWhenDifferent="true")
CopyToOutputDirectory="Always" Copies every build, downstream projects rebuild PreserveNewest
Timestamps or build numbers in AssemblyInfo Every compile has new input Only set versions in CI
Globbing a huge folder (node_modules, test data) Slow evaluation DefaultItemExcludes or Compile Remove

Step 4: Find slow full builds

  • Analysers: third-party analysers on big projects can cost minutes; ReportAnalyzer shows which. Consider running heavy ones only in CI, or only on changed projects.
  • Restore: slow feeds or no NuGet cache in CI.
  • Serial graph: one huge project at the root of the graph limits parallelism; splitting it, or removing unnecessary references, helps.
  • Build servers: CI agents that disable node reuse and the compiler server pay start-up costs repeatedly; for local builds they should stay on.

Production example

A 70-project solution went from 2 to 15 minutes over a few months, and every build recompiled everything. The binary log showed a GenerateVersionInfo target in Directory.Build.targets that wrote VersionInfo.g.cs with the current timestamp in every project. CoreCompile then reran everywhere. A second hotspot was a security analyser taking 40% of compile time. Writing the file only when the version changed, and running that analyser in a nightly CI job instead of every local build, brought full builds back to 3 minutes and no-change builds to 10 seconds.

Best practices, performance and security

  • Treat build time as a metric: track full and no-change build times in CI.
  • Binary logs capture environment variables and property values, which can include secrets and tokens. Don't attach them to public issues, and restrict access in CI artefacts.
  • Keep custom targets few and incremental; review changes to Directory.Build.targets for build-time impact.

🔍 What the interviewer is checking

Probes depth with MSBuild diagnostics and performance thinking.

Green flags: starts with -bl and the Structured Log Viewer; explains how to find why a target ran; knows common incrementality breakers; measures analysers; mentions binlog sensitivity; adds a regression guard.

Red flags: guesses; only suggests bigger machines; has never opened a binary log.

📚 Learn in depth

1. Where build time goes

A dotnet build passes through distinct phases, and each one fails in its own way:

 dotnet build
   |
   +-- restore            NuGet: resolve the graph, download, write project.assets.json
   |                      slow when: no package cache, slow feeds, huge graphs
   |
   +-- evaluation         read every project, import .props/.targets, expand globs
   |                      slow when: globs over node_modules or test data, deep imports
   |
   +-- execution          run targets per project, in parallel across nodes
         |
         +-- ResolveReferences / ResolvePackageAssets
         +-- custom targets (code generation, Exec, npm, ...)   <-- common culprits
         +-- CoreCompile -> Csc task: compiler + analysers + source generators
         +-- copy to output (bin/), dependants pick up the new assembly

The two symptoms in the question point to different causes. A slow full build is about how much work each phase does. A non-incremental build means work is repeated that should have been skipped, and that is almost always one changed input causing a chain reaction.

2. Capturing a binary log

Command What it gives you
dotnet build -bl msbuild.binlog in the current folder: every property, item, import, target, task, message and timing
dotnet build -bl:incremental.binlog A named log, so you can compare two builds
dotnet build -clp:PerformanceSummary Text summary of evaluation, target and task times at the end of the console output
dotnet build -bl -p:ReportAnalyzer=true Adds time per analyser and per source generator to the Csc task in the log
MSBUILDDEBUGENGINE=1 (environment variable) Makes Visual Studio and other hosts write binary logs, into an MSBuild_Logs folder, for builds you can't start from the command line

A binary log is small, lossless and replayable: dotnet build msbuild.binlog -v:diag (or dotnet msbuild msbuild.binlog -v:diag) replays it as a diagnostic text log without building again. Capturing with -bl is cheaper than -v:diag on the console and loses nothing.

3. Reading it in the Structured Log Viewer

The MSBuild Structured Log Viewer (msbuildlog.com: a Windows app, a cross-platform Avalonia app and a web viewer) shows the log as a tree. The views that matter for performance:

View Answers
Timeline Which nodes were busy, what ran in parallel and where the critical path is
Target and task summaries (on the root node) Total time per target and per task across all projects
Search $target CoreCompile, $task Csc, $project Orders.Api, a property name or a file path
Properties and items per project The final values after evaluation, and where each was set
Evaluation node Evaluation time per project, and the import tree

For each target the log records whether it ran and why:

  • Skipping target "CoreCompile" because all output files are up-to-date with respect to the input files. This is what you want to see in a no-change build.
  • Building target "CoreCompile" completely. Input file "obj/.../VersionInfo.g.cs" is newer than output file "obj/.../Orders.Domain.dll". This names the input that broke incrementality.
  • A target with no Inputs and Outputs has no such message: it simply runs every time.

4. How MSBuild decides whether to skip a target

 <Target Name="Gen" Inputs="@(Schema)" Outputs="$(IntermediateOutputPath)Gen.g.cs">

   all Outputs exist  AND  newest Input is older than oldest Output ?
        | yes                                 | no
        v                                     v
   skip the target                      run it (completely, or partially
   ("all outputs up-to-date")           for 1:1 input-to-output mappings)

The check is purely timestamps. That's why a target that rewrites a file with the same content still breaks the chain: the content is identical but the timestamp is new, so CoreCompile sees a newer input. WriteLinesToFile WriteOnlyWhenDifferent="true" and the Copy task's SkipUnchangedFiles="true" exist for exactly this reason.

5. The chain reaction, and how reference assemblies stop it

 incremental.binlog  (second build, nothing changed)

 Orders.Domain
   GenerateVersionInfo   ran    (no Inputs/Outputs, rewrites VersionInfo.g.cs)
   CoreCompile           ran    9.8 s  (input VersionInfo.g.cs is newer)
   reference assembly    CHANGED? -- the timestamp attribute is in the metadata, so yes
        |
        v
 Orders.Application  CoreCompile ran  (reference changed)
 Orders.Api          CoreCompile ran  (reference changed)
 ... every dependant recompiles

The SDK produces reference assemblies (ProduceReferenceAssembly, on by default): a copy of each assembly containing only its public surface. Dependants compile against it, and it is only rewritten when the public surface changes. So editing a method body in Orders.Domain recompiles Orders.Domain only. An assembly attribute containing a timestamp changes the metadata on every build, defeats that optimisation and turns one project's rebuild into a whole-solution rebuild.

6. Common causes and their fingerprints

Cause What you see in the log Fix
Custom target with no Inputs/Outputs Target runs in every build Declare Inputs and Outputs
Generated file rewritten every build CoreCompile ran: generated file is newer WriteOnlyWhenDifferent="true"; stable content locally
Timestamps or build numbers in assembly attributes Every project recompiles, reference assemblies change Set real values only in CI builds
CopyToOutputDirectory="Always" Copies every build; Visual Studio's fast up-to-date check always says "not up to date" PreserveNewest
Globs over large folders Long evaluation time per project DefaultItemExcludes, Compile Remove, or move the folder out
Exec running npm install or similar A long Exec task in every build Inputs on package-lock.json, Outputs on a stamp file
Expensive analysers or source generators Csc is long; ReportAnalyzer shows one package dominating Fix or replace it, or run it in a separate CI job
One huge project at the root of the graph Timeline shows other nodes idle while it compiles Split it, or remove references that aren't needed
Restore every time in CI Long Restore with downloads Cache ~/.nuget/packages keyed on lock files; --no-restore after one restore step

7. Analysers and source generators

With -p:ReportAnalyzer=true, the Csc task in the log lists "Total analyzer execution time" and "Total generator execution time", then the time per analyser and generator assembly and per rule. Options when one dominates:

  • Update it: analyser performance bugs are often fixed in later versions.
  • Turn off its expensive rules in .editorconfig if they add little.
  • Run it in a dedicated CI job (nightly or on pull requests only) instead of every local build, and say so in the build documentation. Keep the SDK security rules in every build.
  • For quick local experiments, -p:RunAnalyzers=false skips analysers entirely. Don't make that the default.

A source generator that isn't incremental (one that uses the old ISourceGenerator API instead of IIncrementalGenerator) can also slow the IDE badly, not just the build.

8. Build servers, nodes and Visual Studio

  • MSBuild keeps worker nodes alive between builds (node reuse), and Roslyn keeps a compiler server (VBCSCompiler) alive, so later builds skip start-up and JIT costs. Keep both on for local builds.
  • dotnet build-server shutdown stops them when they hold stale state, for example after changing a custom task assembly.
  • On ephemeral CI agents they help within a job but not across jobs. Some pipelines turn them off with -nodeReuse:false or UseSharedCompilation=false to avoid locked files; measure before copying that setting locally.
  • Visual Studio has its own fast up-to-date check, which decides whether to call MSBuild at all. If Visual Studio rebuilds a project every time but the command line doesn't, set its up-to-date check logging to Verbose (Tools > Options > Projects and Solutions > SDK-Style Projects) and read the reason in the Build output.
  • In large repos, RestoreUseStaticGraphEvaluation=true makes restore evaluate projects faster, and dotnet build -graph builds from the static project graph. Measure both before adopting them.

9. A repeatable diagnosis

  1. dotnet build -bl:full.binlog (clean or normal build).
  2. dotnet build -bl:incremental.binlog immediately again, with no changes.
  3. In incremental.binlog, search $target CoreCompile. Every one should say "Skipping target". For each one that ran, read the reason and follow the named input back to the target that wrote it.
  4. In full.binlog, open the target and task summaries and the timeline. Note the top five targets and tasks, and whether the timeline shows idle nodes.
  5. If Csc dominates, rebuild with -p:ReportAnalyzer=true -bl and read the analyser breakdown.
  6. If evaluation is long, look at the evaluation node for the slow projects and their globs.
  7. Fix one cause at a time and repeat steps 1 and 2 to prove the improvement.

10. Performance and security

  • Track two numbers in CI: full build time and no-change rebuild time. The second catches incrementality regressions that a full build hides.
  • Binary logs are sensitive. They record the environment variables and properties the build used, command lines, and the project and targets files imported, which can include tokens, feed passwords and connection strings. Store them as restricted CI artefacts, don't attach them to public issues, and use a redaction step (such as the binlogtool redact command) before sharing outside the team.
  • Review changes to Directory.Build.targets and custom tasks for build-time impact, because they run in every project.

11. .NET 11 awareness

.NET 11 (due in November 2026, an STS release with 24 months of support, currently in preview and release candidates) ships a newer MSBuild and compiler. The binary log format and the diagnosis approach stay the same; after an SDK upgrade, compare a binary log from before and after to catch any change in build time.

12. Try it yourself

  1. Add the bad GenerateVersionInfo target from the code example to Directory.Build.targets in a solution with two projects, one referencing the other.
  2. Build twice with -bl:first.binlog and -bl:second.binlog. In the second log, search $target CoreCompile and read why it ran in both projects.
  3. Replace it with the fixed target, build twice again, and confirm both say "Skipping target".
  4. Run dotnet build -clp:PerformanceSummary and compare the summary with the viewer's target summary.
  5. Add an analyser package, build with -p:ReportAnalyzer=true -bl, and find the per-analyser times under the Csc task.
  6. Add <None Include="data.json" CopyToOutputDirectory="Always" /> and see the copy happen on every build; change it to PreserveNewest and build again.

💻 Code

<!-- BAD: runs every build and always rewrites the file, so CoreCompile reruns everywhere -->
<Target Name="GenerateVersionInfo" BeforeTargets="CoreCompile">
  <WriteLinesToFile File="$(IntermediateOutputPath)VersionInfo.g.cs"
                    Lines="[assembly: System.Reflection.AssemblyMetadata(&quot;BuiltAt&quot;, &quot;$([System.DateTime]::UtcNow.ToString('o'))&quot;)]"
                    Overwrite="true" />
  <ItemGroup>
    <Compile Include="$(IntermediateOutputPath)VersionInfo.g.cs" />
  </ItemGroup>
</Target>
<!-- FIXED: stable content locally, real values only on CI, written only when different -->
<PropertyGroup>
  <BuildCommit Condition="'$(BuildCommit)' == ''">local</BuildCommit>
</PropertyGroup>

<Target Name="GenerateVersionInfo" BeforeTargets="CoreCompile">
  <WriteLinesToFile File="$(IntermediateOutputPath)VersionInfo.g.cs"
                    Lines="[assembly: System.Reflection.AssemblyMetadata(&quot;Commit&quot;, &quot;$(BuildCommit)&quot;)]"
                    Overwrite="true"
                    WriteOnlyWhenDifferent="true" />
  <ItemGroup>
    <Compile Include="$(IntermediateOutputPath)VersionInfo.g.cs" />
    <FileWrites Include="$(IntermediateOutputPath)VersionInfo.g.cs" />
  </ItemGroup>
</Target>
# CI guard: a no-change rebuild must stay fast
dotnet build -c Release
start=$(date +%s)
dotnet build -c Release --no-restore -bl:incremental.binlog
elapsed=$(( $(date +%s) - start ))
echo "No-change build: ${elapsed}s"
[ "$elapsed" -lt 60 ] || { echo "Incremental build regression"; exit 1; }

In CI, pass the commit with -p:BuildCommit=$GITHUB_SHA (or $(Build.SourceVersion) in Azure Pipelines). FileWrites tells dotnet clean to remove the generated file.

⚠ Common mistakes

  • Mistake: guessing instead of measuring. "It's the antivirus" or "we need faster agents" might be partly true, but without evidence the real cause stays and the money is wasted. Instead: capture a binary log of a full build and of a no-change rebuild, and let the timings and "why did this run" messages point to the cause.

  • Mistake: reading -v:diag console output. It's huge and flat, so finding why one target ran is slow, and the console log loses structure. Instead: capture with -bl and use the Structured Log Viewer's search and summaries. Replay the binary log as text later if you really need it.

  • Mistake: writing custom targets without Inputs and Outputs. The target runs on every build, and if it writes a file that feeds the compiler, every project recompiles. Instead: declare Inputs and Outputs, and write generated files with WriteOnlyWhenDifferent="true" so the timestamp only changes when the content does.

  • Mistake: putting timestamps or build numbers into assembly attributes in local builds. The metadata changes every build, so reference assemblies change and every dependant recompiles. Instead: use stable values locally and pass the real commit or build number only in CI, for example -p:BuildCommit=$GITHUB_SHA.

  • Mistake: using CopyToOutputDirectory="Always". The file is copied on every build and Visual Studio's fast up-to-date check always rebuilds the project. Instead: use PreserveNewest.

  • Mistake: deleting an expensive analyser without telling anyone. Builds get faster, but the bugs it caught (often security issues) come back unnoticed. Instead: measure it with ReportAnalyzer, update or tune it, or move it to a dedicated CI job, and keep the SDK security rules in every build.

  • Mistake: sharing binary logs publicly. They can contain tokens, feed credentials and connection strings from properties and environment variables. Instead: keep them as restricted CI artefacts, and redact them before sharing outside the team.

  • Mistake: fixing it once without a guard. Build performance decays one target or package at a time, so the regression returns within months. Instead: add a CI job that times a no-change rebuild and fails above a threshold, and track full build time on a dashboard.

  • Mistake: testing only the command line when developers complain about Visual Studio. Visual Studio has its own fast up-to-date check, so it can rebuild projects that dotnet build skips. Instead: turn on verbose up-to-date check logging in Visual Studio and read its reason, or capture a log from the IDE with MSBUILDDEBUGENGINE=1.

🏛 Senior insight: trade-offs

Build performance decays quietly: one custom target, one analyser package, one glob at a time. A senior engineer treats it like production performance: measure with the right tool (binary logs, analyser reports), find the mechanism (an input that changes every build, a serial bottleneck, an expensive analyser), fix the cause, then guard it with a metric in CI. They also weigh trade-offs openly, for example running an expensive security analyser nightly instead of in every inner-loop build, rather than just deleting it.

🔗 Follow-ups the interviewer may ask

  1. When: Candidate talks about targets being skipped or not. Why: Checks the MSBuild fundamentals behind the diagnosis.
    How to answer

    MSBuild is the build engine under dotnet build, Visual Studio and Rider, and a .csproj is an MSBuild file. Properties are named values such as $(Configuration). Items are lists with metadata, such as @(Compile) holding every .cs file. Targets are named steps, ordered with DependsOnTargets, BeforeTargets and AfterTargets. Each target runs tasks, the real units of work, such as Csc to compile or Copy. The build has two phases. Evaluation reads imports, properties and items. Execution runs the targets. Targets that declare Inputs and Outputs are skipped when nothing has changed, which is what makes builds incremental.

  2. When: Candidate finds the culprit in a shared targets file. Why: Shared targets in Directory.Build.targets are a common source of slow builds.
    How to answer

    Directory.Build.props and Directory.Build.targets are ordinary MSBuild files that the SDK imports automatically into every project below them. For each project, MSBuild searches upwards and uses the nearest one it finds. .props is imported early, before the project's own content, so it's for default properties and items: target framework, Nullable, analysers, package metadata. Projects can still override those values. .targets is imported late, after the project, so it's for targets and logic that need final values. Only the nearest file is picked up. To layer them, the inner file imports the outer one with GetPathOfFileAbove. I debug them with dotnet msbuild -preprocess or a binary log.

  3. When: Candidate proposes tracking build time in CI. Why: Scales build performance to repo-wide governance and metrics.
    How to answer

    I'd treat the build as a product, with a contract that lives in files at the repo root:

    • global.json for the SDK.
    • Layered Directory.Build.props/.targets for framework, nullable, analysers and output layout.
    • Directory.Packages.props with transitive pinning for versions.
    • nuget.config with source mapping, plus lock files.
    • .editorconfig for style and analyser severities.

    Project files stay tiny. A small platform group owns those files through CODEOWNERS, and there's a documented exception process for overrides and suppressions. CI uses shared templates that run exactly what developers run locally. The warning policy ratchets stricter over time. SDK, framework and package upgrades arrive as automated, grouped pull requests. I'd measure it all (build time, warnings, drift, time to patch) so governance helps teams instead of just blocking them.

🛠 Real-world problems: how you handle them

A timestamp target makes every build a full rebuild
Situation
A 70-project .NET 10 solution used by four teams. Over a few months, full builds rose from 2 to 15 minutes and every build, even with no changes, recompiled every project.
Signals
Developers complaining about inner-loop speed; CI build time trending up on the dashboard; 'Building Orders.Domain' appearing on every build even with no edits.
Investigation
Captured two consecutive binary logs and opened the second one in the MSBuild Structured Log Viewer. CoreCompile ran in every project with the reason that VersionInfo.g.cs was newer than the output. Traced the file to a GenerateVersionInfo target in Directory.Build.targets that wrote a UTC timestamp every build. A ReportAnalyzer run also showed one security analyser package taking about 40% of compile time.
Resolution
Rewrote the target to write a stable value locally and the commit SHA on CI, with WriteOnlyWhenDifferent. Moved the expensive analyser to a nightly CI job and kept the SDK security rules in every build.
Prevention
A CI job that fails if a no-change rebuild exceeds 60 seconds, review of Directory.Build.targets changes by the platform owners, and a build-time dashboard per week.

How to tell it (STAR)

**Situation:** Our [70-project] solution at [company] had slowed from [2] to [15] minutes, and even no-change builds recompiled everything. **Task:** I volunteered to find the cause, since it was costing every developer [hours] a week. **Action:** I compared binary logs of two consecutive builds and saw CoreCompile rerunning because a shared target wrote a timestamp into generated code every time. I made the target write stable content locally and only when it changed, measured analysers with ReportAnalyzer, and moved one expensive analyser to a nightly job. I added a CI guard on no-change build time. **Result:** Full builds dropped to [about 3] minutes and no-change builds to [about 10] seconds, and the guard has caught [one] regression since.

↩ Usually asked after

More in Build System

Last updated 1 Nov 2026.

🌐 Sangam's Webring

⏪ Previous 🏠 Main Next ⏩
💬 Chat with me