Incident
On Black Friday (US time) 2023, I got a call early in the morning. The site was running discounts of $400–650 USD, and a massive wave of traffic hit all at once — orders were coming in every second, many times the normal volume. The site loaded pages fine, but failed to retrieve real-time inventory from Redis correctly. This left users stuck on a loading spinner while the cart waited for inventory data from Redis, preventing them from adding items to their cart.
Root Cause Analysis
Investigation revealed that three Redis instances were experiencing sync failures. After analyzing hardware performance, we found that these three Redis slave containers were all running inside a nested Hyper-V host. They shared a single NIC, and the Windows host had only 16 GB of RAM allocated — with Docker Desktop capped at 8 GB. Utilization was near 100% across the board.
The conclusion was that insufficient memory was the primary cause of the sync anomaly. As an immediate fix, we shut down one Redis container on that host to free up resources so the remaining containers could operate normally.
Remediation Plan
Short-Term
- Shut down one instance to allow the host to function properly.
- Increase available memory.
Medium-Term
- Replace the resource-heavy nested Hyper-V setup with Nginx.
- Tune Redis cache eviction settings to release memory from unused keys.
Long-Term
- Consider adopting Redis Cluster with sharding.
- Replace Windows Server containers with Linux containers.
- Introduce Kubernetes for automated deployment, scaling, and container management.
Comments