DevOps

PodDisruptionBudgets: Your Kubernetes Outage Insurance

It's Tuesday morning . The platform team starts draining nodes for a Kubernetes upgrade. Sixty seconds later, Slack explodes — the payment service i…

War Story: We Survived a 2-Hour Outage with Redis 8.0 Cluster and Sentinel

At 14:17 UTC on October 12, 2024, our Redis 8.0.2 cluster serving 142,000 writes/sec and 89,000 reads/sec across 12 shards dropped 100% of traffic f…

The Postmortem of a 20-Minute Kafka 3.8 Outage That Delayed 1M Order Messages

On October 12, 2024, a misconfigured Kafka 3.8 broker idempotent write setting caused a 20-minute total outage that delayed 1,047,892 order messages…