The issue lasted about 1,5 hours and affected 80 Pro Apps on production plans (not development nor dedicated) hosted in the US region. Pro Apps on production plans have redundancy in load balancing, routing and web delivery, but this issue was caused by a misconfiguration in our automated DNS setup for a single routing Node. This caused web requests to fail before they could reach our redundant setup. The issue was solved by an on-call admin running a reconfigure command on the automated DNS setup.
What caused the DNS setup to misconfigure remains undetermined. The software has a track record of stability, with no comparable incidents documented. Despite recent extensive updates, the possibility of this happening again cannot be completely dismissed. We are on alert for this and implementing additional sanity checks and monitoring.
We think to have fixed the issue and are now monitoring the situation.
It seems the routing service is misconfigured for so far unknown reasons. It seems to affect all Pro Apps in US that on Production plans (not dedicated and not development).
We are looking into issues with web delivery for Pro Apps.
We’ll find your subscription and send you a link to login to manage your preferences.
We’ve found your existing subscription and have emailed you a secure link to manage your preferences.
We’ll use your email to save your preferences so you can update them later.
Subscribe to other services using the bell icon on the subscribe button on the status page.
You’ll no long receive any status updates from fortrabbit, are you sure?
{{ error }}
We’ll no longer send you any status updates about fortrabbit.