Menu

Intermittent Service Outages: Rolling Regional DNS Outages in the U.S.

Release Date:
October 21, 2016

Update Mon, Oct 24 – 9:20a MT: Dyn is reporting that the DDoS attack against their DNS infrastructure has been mitigated. We will conduct a root cause analysis and update customers with our findings and any plans for changes in advance of the holiday season.

===

Update: Fri, Oct 21 – 3:54p MT: Search query volumes have returned to normal levels. DNS resolution continues to be stable and response times are within normal levels.  We are now advising customers to rollback any temporary mitigation changes that were activated today.

We continue to monitor the situation closely.

===

Update: Fri, Oct 21 – 3:08p MT: We’re seeing an improvement in DNS resolution across the U.S. and Europe via DynDNS.  Search query volumes have returned to normal levels.  We continue to monitor the situation closely.

===

Our upstream DNS provider, Dyn, is being affected by rolling DDoS attacks today. This is impacting the resolution of SearchSpring hostnames. The disruption is also affecting Netflix, Spotify, Amazon, Twitter, and many others as well.

 

Affected hostnames:

  • api.searchspring.net
  • cdn.searchspring.net
  • manage.searchspring.net

Dyn is actively working to mitigate the attack and reduce its impact. Some customers are reporting trouble accessing SearchSpring services.

 

Real-time updates regarding the issue here:

DNS updates from Dyn: https://www.dynstatus.com/

Twitter mentions: https://twitter.com/search?f=news&vertical=news&q=ddos&src=typd

Map of affected locations: http://downdetector.com/status/level3/map/

Internet Health Report (routing and latency): http://www.internethealthreport.com/

 

Temporary Alternative Hostnames:

api.searchspring.net -> api.searchspring.io

api.searchspring.net -> api.searchspring.io

cdn.searchspring.net -> cdn.searchspring.io

The hostnames above are temporary alternatives that route around Dyn. If you activate an alternative hostname, please be prepared to undo this action after the incident is over. Dyn plays a critical part in our ability to manage uptime and failover for backend services and by routing around Dyn, we are giving up the ability to mitigate backend issues with our system. We are committed to providing our customers with workarounds during extenuating circumstances, which is why we are providing these alternatives.

 

Overview of the Issue:

Some SearchSpring customers and their shoppers may be experiencing slow search response times and temporary outages. This is due to a large-scale outage of a large domain name server (DNS) service. This issue is not limited to only SearchSpring; it is also causing outages for Shopify, Twitter, Github, and many other major web services.

 

Further Details:

DynDNS, a major DNS service, provides web browsers with information about domain names so they can look up IP addresses for web servers. DynDNS has experienced multiple sporadic denial-of-service attacks this morning, which means that their service is receiving too many lookup requests to respond to all of them in a timely manner. When web browsers are unable to look up DNS information for the SearchSpring domain names, the browsers are not able to access SearchSpring and shoppers are not able to view search results. (This issue is also impacting several other major web services, including Shopify, Twitter, and Github.)

DynDNS has already resolved this issue in some regions and is working to resolve any further issues that arise. SearchSpring servers are still operating at full capacity and experiencing normal response times of 70-150 milliseconds.

Our Industry Guides