An API endpoint is slow. How do you use the Visual Studio Performance Profiler, dotnet-counters and dotnet-trace to find and fix the cause?

1. The method: measure, locate, fix, verify

 1. MEASURE   APM (Application Insights): which endpoint, p95/p99, throughput,
              dependency breakdown. Record a baseline. Agree a target.
      |
 2. LOCATE    Is it CPU, waiting, or GC?
      |         dotnet-counters (cheap, live)  --> narrows the category
      |         profiler / dotnet-trace         --> finds the code
      |
 3. FIX       one change at a time, the biggest cause first
      |
 4. VERIFY    same load, same data, compare with the baseline (p95 AND throughput)
      |
 5. GUARD     load test with thresholds in the pipeline, alerts, benchmark

Skipping step 1 is the most common failure: people profile whatever they suspect and optimise code that isn't on the slow path.

2. Where the time goes in one request

 request --> Kestrel --> middleware --> endpoint --> service --> EF Core --> SQL Server
               |            |             |           |            |
             CPU          CPU           CPU       await I/O    network + query

 total latency = CPU in your code
               + time waiting (I/O, locks, thread pool queue)
               + GC pauses
               + network round trips (x number of calls)

3. Sampling versus instrumentation

Sampling only sees threads that are running. A request that is slow because it waits on SQL barely shows up in CPU Usage. That is why you switch to the Database or Async tools when CPU looks quiet.

4. Reading a flame graph

 |------------------- Kestrel request processing ------------------|
 |---------- OrdersEndpoint.Search ----------|--- JSON serialise ---|
 |-- EF Core query --|--- Map to DTO (x180) ---|
                     |-- string.Format --|

• Each bar is a frame; the bars above it are the methods it called.
• Width is the share of samples, so wide bars are where time goes.
• Look for wide bars that you own. A wide string.Format under your mapper is actionable; a wide Kestrel frame on its own is not.
• Repeated narrow towers of the same method usually mean it is called many times (an N+1 pattern or a loop).

5. The Visual Studio Performance Profiler tools

Open with Debug > Performance Profiler (Alt+F2). Run against a Release build, without the debugger.

The Diagnostic Tools window shown while debugging is useful for a rough picture, but the debugger changes timings, so don't draw conclusions from it.

6. dotnet-counters: the first look in any environment

dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting

In .NET 9 and later the System.Runtime meter uses OpenTelemetry-style names. The ones that matter most:

These are the same metrics you can export through OpenTelemetry to Azure Monitor or Prometheus, so you can watch them permanently rather than only during an incident.

7. dotnet-trace: CPU samples and events

dotnet-trace collect -p <pid> --duration 00:00:00:30 --format Speedscope

• The default collection includes CPU samples and common runtime events.
• .nettrace files open in Visual Studio or PerfView; --format Speedscope gives a file for speedscope.app.
• Keep captures short (30 to 60 seconds) under real load.
• In Azure App Service, the Application Insights Profiler captures traces on demand or on triggers. In Kubernetes, dotnet-monitor can collect traces through an HTTP API.

8. The thread pool signal in detail

 each request does Task.Delay(200).Wait()   <-- blocks a thread for 200 ms

 load: 100 requests/second
  needs ~20 threads blocked at any time
  thread pool starts with ~(CPU count) threads and injects more slowly

 result: queue length climbs, latency climbs far beyond 200 ms,
         CPU stays moderate

The fix is await Task.Delay(200), which releases the thread while waiting. Raising ThreadPool.SetMinThreads hides the symptom and uses more memory, but the blocking is still there.

9. The usual causes and fixes

10. Latency and throughput together

A change can make one request faster but reduce how many requests the service handles, for example by adding parallel calls that use more threads or connections. Always measure both p95 latency and requests per second under the same load before and after, and record the results with the change so later work has a baseline.

11. Security

Traces and profiler sessions can capture SQL text with parameters, URLs, headers and payloads. Treat trace files like logs that may contain personal information: store them with restricted access and delete them after analysis.

12. .NET 11 awareness

Each .NET release improves the JIT, GC and libraries, so re-baseline after upgrading. Try .NET 11 previews in a benchmark branch, not by profiling production.

13. Try it yourself

1. Run the code example's API in Release and call /orders/slow and /orders/fast in a loop.
2. Profile with CPU Usage and Database, and compare the number of queries per request.
3. Run dotnet-counters monitor while sending concurrent requests to /report/blocking and watch the thread pool queue length and thread count.
4. Change it to await Task.Delay(200) and repeat.
5. Capture dotnet-trace for 30 seconds under load and open it in speedscope to find the widest frames you own.
6. Record p95 and requests per second before and after each change.

A minimal API with three deliberate problems to profile (EF Core with SQLite so it runs anywhere):

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.0" />
  </ItemGroup>
</Project>

using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddDbContext<ShopDb>(o => o.UseSqlite("Data Source=shop.db"));
var app = builder.Build();

using (var scope = app.Services.CreateScope())
{
    var db = scope.ServiceProvider.GetRequiredService<ShopDb>();
    db.Database.EnsureCreated();
    if (!db.Customers.Any())
    {
        for (var i = 0; i < 200; i++)
        {
            var customer = new Customer { Name = $"Customer {i}" };
            customer.Orders.Add(new Order { Total = i * 10m });
            db.Customers.Add(customer);
        }
        db.SaveChanges();
    }
}

// N+1: one query for orders, then one query per order for its customer
app.MapGet("/orders/slow", async (ShopDb db) =>
{
    var orders = await db.Orders.ToListAsync();
    var result = new List<object>();
    foreach (var order in orders)
    {
        var customer = await db.Customers.SingleAsync(c => c.Id == order.CustomerId);
        result.Add(new { order.Id, order.Total, Customer = customer.Name });
    }
    return result;
});

// One query with a projection
app.MapGet("/orders/fast", async (ShopDb db) =>
    await db.Orders
        .Select(o => new { o.Id, o.Total, Customer = o.Customer.Name })
        .ToListAsync());

// Sync-over-async: blocks a thread pool thread for each request
app.MapGet("/report/blocking", () =>
{
    Task.Delay(200).Wait();
    return Results.Ok("done");
});

app.Run();

public sealed class ShopDb(DbContextOptions<ShopDb> options) : DbContext(options)
{
    public DbSet<Customer> Customers => Set<Customer>();
    public DbSet<Order> Orders => Set<Order>();
}

public sealed class Customer
{
    public int Id { get; set; }
    public required string Name { get; set; }
    public List<Order> Orders { get; } = [];
}

public sealed class Order
{
    public int Id { get; set; }
    public decimal Total { get; set; }
    public int CustomerId { get; set; }
    public Customer Customer { get; set; } = null!;
}

CLI diagnostics against the running process:

dotnet tool install --global dotnet-counters
dotnet tool install --global dotnet-trace

dotnet-counters monitor -p 12345 --counters System.Runtime,Microsoft.AspNetCore.Hosting
dotnet-trace collect -p 12345 --duration 00:00:00:30 --format Speedscope