Problem
Every team ran its own Prometheus server. Cross-cluster queries were impossible, and metrics were deleted after 15 days.
Solution
- Long-term retention in object storage with Thanos Sidecar + Store Gateway
- Global queries across all regions from a single Grafana
- Cutting the load time of heavy dashboards by 70% with recording rules
Result
Getting to the right data during incidents became faster; MTTR dropped by 42% and capacity planning became possible with 13 months of data.