Structured logs, metrics and traces, SLOs with alerts that page.
Spec — Observability
Status: draft · Created: 2026-11-10
1. Context and goal
Engineers see how the service behaves in production, find the cause of a failure quickly, and are paged only when users are affected.
2. Functional requirements (EARS)
- FR-1: THE SYSTEM SHALL write logs as JSON lines with time, level, service, message and trace id.
- FR-2: THE SYSTEM SHALL never log passwords, tokens or full payment card numbers, checked by a test on the log output.
- FR-3: THE SYSTEM SHALL export request rate, error rate and latency histograms for every endpoint, scraped at least every 30 seconds.
- FR-4: THE SYSTEM SHALL propagate a trace context across every service call and queue message, and keep traces for at least 7 days.
- FR-5: THE SYSTEM SHALL define SLOs for availability and latency per user-facing service, such as 99.9% of requests succeeding over 30 days.
- FR-6: WHEN an SLO error budget burns at more than 14 times the sustainable rate over 1 hour, THE SYSTEM SHALL page the on-call engineer.
- FR-7: WHEN a slower burn would exhaust the budget within 3 days, THE SYSTEM SHALL open a ticket instead of paging.
- FR-8: THE SYSTEM SHALL link every alert to a runbook and a dashboard of the affected service.
3. Out of scope
- Business analytics.
- Profiling in production.