Observability: Logs, Metrics and Traces

Observability is the ability to understand what a running system is doing, and why, from the data it emits. It rests on three kinds of telemetry: metrics (numbers over time), logs (records of events) and traces (the path of one request through many services). Good observability is what turns a 3-hour outage into a 10-minute one.

The three pillars

Distributed tracing

In a microservice system, one user request may touch ten services. A trace ID generated at the edge is passed along on every call (the W3C traceparent header); each service records spans with start time, duration and attributes. The resulting trace shows exactly which hop was slow or failed.

OpenTelemetry has become the standard for producing traces, metrics and logs in a vendor-neutral way, with automatic instrumentation for common frameworks.

What to measure: RED and USE

• RED for every request-driven service: Rate (requests per second), Errors (failed requests), Duration (latency percentiles: p50, p95, p99). See latency vs throughput.
• USE for every resource (CPU, memory, disks, connection pools, queues): Utilisation, Saturation (queued work), Errors.
• Business metrics: orders per minute, sign-ups, payment success rate. These catch problems that technical metrics miss.

Logging well

• Use structured logs (key-value or JSON) so they can be filtered and aggregated, not free text.
• Include the trace ID and request ID in every log line, so you can jump from a trace to its logs.
• Use levels deliberately; log decisions and failures, not every line of execution.
• Never log secrets, passwords, tokens or full payment data; redact personal data.
• Control cost with sampling for high-volume debug logs and retention tiers (hot for days, archived for months).

Cardinality

Every unique combination of metric labels creates a separate time series. Labels like status_code or region are fine; labels like user_id or request_id explode into millions of series and can overwhelm a metrics system. High-cardinality detail belongs in logs and traces, not metric labels. See time-series databases.

Alerting

• Alert on symptoms users feel (error rate, latency, failed checkouts), not on every cause (CPU at 80%). Causes go on dashboards.
• Tie alerts to SLOs and use burn-rate alerts: page when the error budget is being consumed fast enough to exhaust it soon. See availability and SLOs.
• Every page should be actionable and link to a runbook; noisy alerts train people to ignore them.