Web delivery issues with Pro Apps in US

Resolved
Resolved

The issue lasted about 1,5 hours and affected 80 Pro Apps on production plans (not development nor dedicated) hosted in the US region. Pro Apps on production plans have redundancy in load balancing, routing and web delivery, but this issue was caused by a misconfiguration in our automated DNS setup for a single routing Node. This caused web requests to fail before they could reach our redundant setup. The issue was solved by an on-call admin running a reconfigure command on the automated DNS setup.

What caused the DNS setup to misconfigure remains undetermined. The software has a track record of stability, with no comparable incidents documented. Despite recent extensive updates, the possibility of this happening again cannot be completely dismissed. We are on alert for this and implementing additional sanity checks and monitoring.

Avatar for
Recovering

We think to have fixed the issue and are now monitoring the situation.

Avatar for
Updated

It seems the routing service is misconfigured for so far unknown reasons. It seems to affect all Pro Apps in US that on Production plans (not dedicated and not development).

Avatar for
Investigating

We are looking into issues with web delivery for Pro Apps.

Avatar for
Began at:

Affected components
  • US
    • Pro Apps
      • Web delivery
      • Object Storage
      • Memcache
      • Worker