The Black Friday incident where our on-call saved $340k

Black Friday, 3:17am UTC. The alert fired on our monitoring system for a client running a Magento B2B store. Traffic had spiked 8x their normal load as their sale went live. Here’s the full timeline.

3:17am — First alert

Response time on checkout exceeded our 3-second threshold. Our on-call engineer was paged. Initial assessment: traffic spike causing database query saturation. Normal for Black Friday — but the queue was growing faster than expected.

3:24am — Escalation

Response time crossed 8 seconds. The checkout was functionally unusable. Database slow query log showed one query running 12 seconds repeatedly: a full table scan on the order history table with no index on the date column. This query ran on every checkout page load to calculate “you’ve spent X with us” for the loyalty display widget.

3:31am — Mitigation

We disabled the loyalty widget at the template level — a 30-second change. Response time immediately dropped to 1.4 seconds. Checkout was functional. Revenue flow resumed.

4:15am — Root cause confirmed

The loyalty widget query had worked fine under normal load. At 8x traffic with concurrent checkouts, the unindexed query created a lock cascade on the orders table. We added the index to staging, tested, deployed to production without downtime.

4:47am — Full service restored

Loyalty widget re-enabled with the indexed query. All checkout flows normal. The outage window was approximately 87 minutes.

The $340k figure

The client’s average revenue rate during their Black Friday sale was approximately $235/minute (based on the first 15 minutes of data before the incident). 87 minutes × $235 = approximately $20k in direct lost revenue. The $340k figure is our estimate of total sale-period revenue preserved — because the sale ran for 24 hours and recovering the checkout before the peak European shopping window opened was the critical factor. We don’t take credit for all of it, but we do know the checkout being down during peak hours would have cost significantly more.

What changed after

Automated load testing before every major sale event. Query performance monitoring with alerts on slow queries. The loyalty widget specifically now runs on a cached count, updated every 60 minutes, rather than a live query. Staging environment that mirrors production load characteristics for pre-event testing.