On September 12, 2024, we began an internal load test that happened after an experimental change to the database infrastructure was made. The two together inadvertently led to internal connection exhaustion, due to a shared dependency with our core services. This affected the Shopify admin and checkout services in one region of our infrastructure.

The issue was mitigated by stopping the load tests and reverting the problematic database changes. We locked our load testing tooling, and put mitigations in place to ensure this did not recur.

An internal root cause analysis meeting was conducted on September 13 2024, to understand how this issue occurred and how we can eliminate future recurrence.

Shopify is taking the following remediation steps to ensure our systems respond in more resilient ways, in order to minimize impact and reducing the duration of the such issues:
- We are investigating separation of resources across load testing and production environments without affecting admin and storefront performance, including checkout.
- We have reduced the impact that similar database changes can have in the short term.