Interruption in service on all stores
Incident Report for Shopify
Postmortem

Our mission is to make commerce better for everyone by providing a reliable platform for you and your customers. Last Sunday we fell short of that goal as we experienced a full platform outage. We apologize for the impact that this outage had on your business. Your trust in us to run your business is something that we take very seriously. What follows is a description of what happened.

On June 2nd, 2019 between 18:48 UTC and 21:58 UTC Shopify was down for all merchants due to an outage in Google Cloud’s global network fabric. This caused an outage for all merchants and customers when browsing, checking out, or managing their stores. At 21:58 UTC, Shopify was operating at 55% availability, gradually recovering until 23:00 UTC when Shopify returned to full, normal operations.

Shopify runs across multiple regions and geographies. Running in multiple regions shields the Shopify platform from events such as large technical failures and natural disasters as each region is isolated from the others with the exception of some shared components, which includes part of the networking fabric.

A store is served out of one region at a time and we have developed advanced tools that move stores between regions in the face of an outage. We can move shops without any downtime. This allows for quick recovery for single region outages. While regional outages are infrequent, they do happen and we continuously invest in our platform to protect against them. For example, we routinely isolate regions in the anticipation of hurricanes that may impact a region.

Our multi-region architecture is built on the industry standard assumption that outages spanning multiple regions are extremely unlikely, and on June 2nd, for the first time in Shopify's history, we saw a multi-region outage of this nature. During the recovery not all regions came back at the same time. We quickly took advantage of our multi-region architecture and began moving stores to healthy regions to get merchants online as quickly as possible.

As uncommon as they are, a core, continental Internet infrastructure outage is a risk for all SaaS providers. Despite this multi-region outage, our architecture continues to be a resilient foundation for Shopify. We remain confident in our partnership with Google Cloud and that they will improve their infrastructure as a result of this outage. Follow their investigation here. Over the coming years we plan to expand into more regions worldwide to increase resiliency and support our merchants’ global expansion.

Jean-Michel Lemieux
CTO

Posted 21 days ago. Jun 04, 2019 - 14:55 EDT

Resolved
All systems are now fully operational. We recognize and apologize for the stress, concern and impact this outage had on your business. In the coming days we will be working to fully understand how this widespread Internet infrastructure failure affected our platform.
Posted 23 days ago. Jun 02, 2019 - 22:15 EDT
Update
All storefronts and checkouts have recovered. Some stores continue to be unable to purchase shipping labels from the fulfillment page in the Shopify admin. We are continuing to monitor to ensure all other systems are operating normally.
Posted 23 days ago. Jun 02, 2019 - 21:47 EDT
Update
All storefronts and checkouts have now recovered. Some stores are temporarily unable to purchase shipping labels from the fulfillment page in the Shopify admin. Shipping options are working normally for customers at checkout -- only the purchase of shipping labels from within the admin is impacted, and we are investigating this. We are continuing to monitor to ensure all other systems are operating normally.
Posted 23 days ago. Jun 02, 2019 - 20:47 EDT
Monitoring
All stores have now recovered. We are continuing to monitor to ensure our systems are operating normally. For more details see the status page for our cloud provider.
Posted 23 days ago. Jun 02, 2019 - 20:12 EDT
Update
Almost all stores have now recovered. We are continuing to work on restoring service for customers using our Canadian infrastructure. For more details see the status page for our cloud provider.
Posted 23 days ago. Jun 02, 2019 - 19:57 EDT
Update
Most stores have now recovered. We are continuing to work with our cloud service provider to resolve this incident for remaining stores.

For more details see the status page for our cloud provider.
Posted 23 days ago. Jun 02, 2019 - 19:50 EDT
Update
Most stores have now recovered. We are continuing to work with our cloud service provider to resolve this incident for remaining stores.

For more details see the status page for our cloud provider.
Posted 23 days ago. Jun 02, 2019 - 19:14 EDT
Identified
Stores are continuing to recover. We are continuing to work with our cloud service provider to resolve this incident for remaining stores.
Posted 23 days ago. Jun 02, 2019 - 18:40 EDT
Update
We are seeing a partial recovery of storefronts and checkouts on some stores. We are continuing to work with our cloud service provider to get this resolved.
Posted 23 days ago. Jun 02, 2019 - 18:11 EDT
Update
We are continuing to work with our cloud service providers to get this resolved.
Posted 23 days ago. Jun 02, 2019 - 17:47 EDT
Update
Our team is still working with our cloud service providers to resolve the issue.
Posted 23 days ago. Jun 02, 2019 - 17:32 EDT
Update
We are continuing to work with our cloud service providers to get this resolved.
Posted 23 days ago. Jun 02, 2019 - 17:17 EDT
Update
Our team is still working with our cloud service providers to resolve the issue.
Posted 23 days ago. Jun 02, 2019 - 17:00 EDT
Update
Our team is still working with our cloud service providers to resolve the issue.
Posted 23 days ago. Jun 02, 2019 - 16:45 EDT
Update
We are continuing to work with our cloud service providers to get this resolved.
Posted 23 days ago. Jun 02, 2019 - 16:26 EDT
Update
We are continuing to investigate this issue.
Posted 23 days ago. Jun 02, 2019 - 15:38 EDT
Update
We are currently experiencing connectivity issues through the entire platform. Most services are inaccessible or with degraded performance. We are working with our cloud service providers to get this resolved as soon as possible.
Posted 23 days ago. Jun 02, 2019 - 15:27 EDT
Update
We are continuing to investigate this issue.
Posted 23 days ago. Jun 02, 2019 - 15:02 EDT
Investigating
Some stores are currently inaccessible. We are investigating and will keep you updated.
Posted 23 days ago. Jun 02, 2019 - 14:56 EDT
This incident affected: Admin, Checkout, Reports and Dashboards, Storefront, API & Mobile, Support, Third party services, and Point of Sale.