How do firewalls, security groups and network policies work? A .NET service cannot reach a dependency, or its connections drop after being idle. How do you troubleshoot it?

1. What "timeout", "refused" and "reset" look like on the wire

Timeout:   SYN ────► (dropped)        retry SYN ────► (dropped) ...  → SocketException TimedOut
Refused:   SYN ────► host
           ◄──── RST (no listener)                                    → ConnectionRefused
Reset:     established ... idle ... firewall forgets flow
           data ────► middlebox / server: unknown connection
           ◄──── RST                                                   → ConnectionReset

Some firewalls send RST instead of dropping ("reject" vs "drop"), so refused can also come from a firewall. Check the source IP of the RST in a capture.

2. Idle timeouts

TCP keep-alive by default on Linux starts after 2 hours, far longer than these. Options: shorter pool idle timeouts in clients, application-level pings, or TCP keep-alive with shorter intervals (SocketsHttpHandler.KeepAlivePingDelay is for HTTP/2 pings; for raw sockets set SocketOptionName.TcpKeepAliveTime).

3. Kubernetes NetworkPolicy

• By default all pods can talk to all pods. Once any policy selects a pod for ingress (or egress), only explicitly allowed traffic passes for that direction.
• A common outage: adding a default-deny egress policy and forgetting DNS (UDP/TCP 53 to CoreDNS), so every name lookup times out.
• Enforcement needs a CNI that supports it (Calico, Cilium, Azure NPM/Cilium).

4. Egress control

Many organisations force outbound traffic through a central firewall with FQDN allow-lists. New dependencies (a SaaS API, a package feed, a telemetry endpoint) then fail with timeouts until rules are added. Keep a documented list of required egress endpoints per service.

5. Tools

nc -vz db.internal 1433                 # Linux/macOS
curl -v telnet://db.internal:1433        # if nc is missing
Test-NetConnection db.internal -Port 1433  # Windows PowerShell
tcpdump -ni any host 10.0.2.4 and port 1433
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash   # debugging pod with tools

6. Try it yourself

1. In a test cluster, apply a default-deny egress NetworkPolicy to a namespace and watch DNS lookups fail; then allow port 53 to kube-dns.
2. Use iptables -A INPUT -p tcp --dport 8080 -j DROP vs -j REJECT on a test VM and compare the client errors.
3. Hold an idle HttpClient connection through a NAT with a short idle timeout and observe the reset on the next request.

// Keep pooled connections shorter-lived than middlebox idle timeouts
builder.Services.AddHttpClient<PartnerClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        PooledConnectionIdleTimeout = TimeSpan.FromMinutes(2),  // below a 4-minute NAT idle timeout
        PooledConnectionLifetime = TimeSpan.FromMinutes(10),
        ConnectTimeout = TimeSpan.FromSeconds(5)               // fail fast on dropped SYNs
    })
    .AddStandardResilienceHandler();                            // retries idempotent requests with backoff

// Raw socket: enable TCP keep-alive with short intervals (.NET Core 3.0+)
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, 60);      // seconds idle
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, 15); // between probes
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, 4);
await socket.ConnectAsync("sftp.partner.com", 22, ct);

# NetworkPolicy: default-deny egress but allow DNS and the database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: orders-egress, namespace: orders }
spec:
  podSelector: { matchLabels: { app: orders-api } }
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector: {}
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    - to:
        - ipBlock: { cidr: 10.20.4.0/24 }   # database subnet
      ports: [{ protocol: TCP, port: 1433 }]