.NET Backend Observability: Telemetry Integration & Performance Optimization

Client

E-commerce platform with high-load .NET 8 backend

Challenge

The .NET backend had no observability β€” no distributed tracing, no centralized metrics, no structured logging. Finding slow requests required manual log digging. The team needed full OpenTelemetry integration: trace propagation from Ingress through .NET to downstream services (PostgreSQL, Redis, RabbitMQ, HTTP upstreams), automatic slow-query detection, and a path to optimize P99 latency.

Solution

1. OpenTelemetry Integration in .NET 8
  • Added OpenTelemetry NuGet packages: OpenTelemetry.Extensions.Hosting, Instrumentation.AspNetCore, Instrumentation.Http, Instrumentation.EntityFrameworkCore, Instrumentation.StackExchangeRedis, Instrumentation.Runtime, Instrumentation.Process, Exporter.OpenTelemetryProtocol
  • Configured ActivitySource with AspNetCore, HttpClient, EF Core, Redis instrumentations
  • Enabled W3C traceparent propagation; added middleware to return X-Trace-Id in response headers
  • Exported via OTLP/gRPC to OTel Collector (:4317)
2. Infrastructure Layer (Kubernetes / Nginx / OTel Collector)
  • Nginx/Ingress: injected traceparent/tracestate headers for W3C context propagation
  • K8s Deployment: environment variables for OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, sampling config (parentbased_traceidratio at 20%)
  • OTel Collector: Tail-based sampling processor β€” always capture errors (status_code: ERROR), slow requests (>1s latency), probabilistic 5% for the rest; drop health checks
3. Methodology for Finding Slow Requests
  • Top-N Slow Endpoints (P95/P99): Grafana dashboards with histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, http_route))
  • Flame Graphs & Span Waterfall: Analyze trace breakdown β€” Middleware β†’ Controller β†’ EF Core β†’ Downstream API β†’ Serialization
  • Runtime Metrics: Monitor ThreadPool queue length, Gen 2 GC frequency, allocations to detect sync-over-async blocking
  • Live Profiling: dotnet-trace + Speedscope/PerfView for CPU sampling on production under load
4. Optimization Checklist Applied
CategoryFixes Applied
DatabaseAdded missing indexes (EXPLAIN ANALYZE), eliminated N+1 with .Include()/.Select(), .AsNoTracking() for read-only, tuned connection pooling
I/O & NetworkFull async/await with CancellationToken, removed .Result/.Wait()
Allocations & GCArrayPool<T>, Memory<T>, ReadOnlySpan<T>, streaming JSON to Response.BodyWriter, enabled Server GC
CachingRedis distributed cache (FusionCache), OutputCaching for HTTP responses
HTTP ClientsIHttpClientFactory / singleton SocketsHttpHandler with PooledConnectionLifetime=15m

Technologies

.NET
.NET 8
OpenTelemetry
OpenTelemetry
Kubernetes
Kubernetes
Prometheus
Prometheus
Grafana
Grafana
Jaeger
Jaeger
PostgreSQL
PostgreSQL
Redis
Redis
RabbitMQ
RabbitMQ
Helm
Helm

Results

βœ… Full trace coverage: 100% of requests have traceparent, X-Trace-Id returned to clients
βœ… P99 latency reduced: from 2.3s β†’ 420ms (82% improvement) after optimizing top-5 endpoints
βœ… Error detection: Tail-sampling captures 100% of 5xx errors and >1s requests even at 20% sample rate
βœ… Proactive alerting: P99 > SLA alerts + error-rate spikes configured in Grafana/Alertmanager
βœ… Database optimization: N+1 eliminated, missing indexes added, connection pool tuned β€” EF Core spans down 65%
βœ… ThreadPool health: Queue length near zero under peak load; sync-over-async eliminated
βœ… Load test validated: k6 soak test confirms stable P99 under 2x expected RPS

Architecture

graph LR A[Ingress / Nginx] -->|W3C traceparent| B[.NET Backend] B -->|OTLP gRPC :4317| C[OTel Collector] C --> D[VictoriaMetrics / Prometheus] C --> E[Jaeger / Tempo] C --> F[Loki] B --> G[(PostgreSQL)] B --> H[(Redis)] B --> I[RabbitMQ] B --> J[HTTP Upstreams]

Duration

2 weeks (integration + collector config + optimization sprint + load testing)

Cost

from $2,000