This is one of the cases we get called in on most often: a store that looks like it’s working. Orders are coming in, support is quiet, conversion rate looks normal. But revenue is down and no one can explain why.
In this case, a fashion label on WooCommerce noticed their revenue was about 14% below forecast for six weeks running. Not a crash — a slow bleed. Their checkout completion rate in GA4 looked normal. Their payment gateway dashboard showed normal approval rates. Nobody flagged it.
When we got access, the first thing we did was check the order status distribution in WooCommerce. We saw an unusually high ratio of “pending payment” orders that were never transitioning to “processing.” These weren’t abandoned carts — the customer had reached the payment step, submitted, and the order was created, but the gateway callback was timing out before it could update the order status.
The root cause: a hosting provider had silently changed their outbound request timeout from 30s to 8s as part of a “performance optimization.” The payment gateway’s webhook was taking 9-12 seconds on average. Every single one was timing out.
We patched it in three steps: increased the timeout threshold, added a retry queue for failed webhooks, and implemented a reconciliation job that catches any orders stuck in pending for more than 15 minutes and re-queries the gateway directly.
We also rebuilt their order status monitoring so this class of issue surfaces as an alert within 30 minutes rather than six weeks.
$180k recovered in the 30-day window after the fix. That’s money that was sitting in a pending queue — some of it we were able to manually process, the rest was refunded and recaptured.
“They found the leak in 20 minutes. The old agency spent two months on it.” — Client, fashion label
The client called us on day two of the outage. They’d attempted a Magento 2.4.5 → 2.4.7 upgrade in-place on production, without a proper staging run, on the Friday before a major sale.
The upgrade had partially completed. The database schema was in a hybrid state — some tables updated for 2.4.7, some still on 2.4.5 structure. The cache was corrupted. Several third-party extensions were incompatible with the new version and throwing fatal errors on checkout.
The store was completely non-functional. 404 on every product page, 500 on checkout, admin panel partially accessible.
Day one: we rolled back to the last full backup (18 hours old — they’d been running on autopilot backups, not pre-upgrade snapshots) and got the store back online on 2.4.5. Revenue loss was severe but stopped.
Day two: we audited every third-party extension for 2.4.7 compatibility. Three were blocking: a custom shipping calculator, a B2B quote module, and a legacy loyalty integration. Two had compatible versions from the vendor; one needed a custom patch.
Days three through seven: built a proper staging environment mirroring production, ran the upgrade in staging, identified and resolved every conflict, ran UAT with the client’s team.
Day eight: zero-downtime cutover using read-replica strategy — synced data to staging environment, pointed DNS, migrated remaining delta in a 12-minute window at 2am local time.
The upgrade was successful. We also set them up with automated pre-upgrade snapshot backups, a proper staging pipeline, and a policy that no production changes happen without a staging run first.
Total downtime including the original botched attempt: 4 days. Estimated revenue loss: ~$85k. Not recoverable, but the store is now running on a modern, maintained version of Magento with a sane deployment process.
Black Friday, 3:17am UTC. The alert fired on our monitoring system for a client running a Magento B2B store. Traffic had spiked 8x their normal load as their sale went live. Here’s the full timeline.
Response time on checkout exceeded our 3-second threshold. Our on-call engineer was paged. Initial assessment: traffic spike causing database query saturation. Normal for Black Friday — but the queue was growing faster than expected.
Response time crossed 8 seconds. The checkout was functionally unusable. Database slow query log showed one query running 12 seconds repeatedly: a full table scan on the order history table with no index on the date column. This query ran on every checkout page load to calculate “you’ve spent X with us” for the loyalty display widget.
We disabled the loyalty widget at the template level — a 30-second change. Response time immediately dropped to 1.4 seconds. Checkout was functional. Revenue flow resumed.
The loyalty widget query had worked fine under normal load. At 8x traffic with concurrent checkouts, the unindexed query created a lock cascade on the orders table. We added the index to staging, tested, deployed to production without downtime.
Loyalty widget re-enabled with the indexed query. All checkout flows normal. The outage window was approximately 87 minutes.
The client’s average revenue rate during their Black Friday sale was approximately $235/minute (based on the first 15 minutes of data before the incident). 87 minutes × $235 = approximately $20k in direct lost revenue. The $340k figure is our estimate of total sale-period revenue preserved — because the sale ran for 24 hours and recovering the checkout before the peak European shopping window opened was the critical factor. We don’t take credit for all of it, but we do know the checkout being down during peak hours would have cost significantly more.
Automated load testing before every major sale event. Query performance monitoring with alerts on slow queries. The loyalty widget specifically now runs on a cached count, updated every 60 minutes, rather than a live query. Staging environment that mirrors production load characteristics for pre-event testing.