Deployment issue in EU
Incident Report for fortrabbit

There was a long downtime, yesterday evening for the deployment service (SSH/Git) for the whole EU area, around 4 hours, connected to a scheduled maintenance done yesterday. Luckily Sunday evening/night is not prime time when it comes to deployment — and it was only deployment, not web delivery — still that was more downtime, as we are comfortable with.

The technical issue was quite complex, as usual in such cases. A simple explanation goes like this: There was a complication when restarting the service after upgrade, resulting in an unstable state. We immediately cloned the service, but that took a little time and after that was finished, numerous other problems appeared.

We — of course — learned a lot and we are going to implement further steps in hardening the platform, by technical improvements and better procedures protocol.

We have probably focused too much on the web delivery part — as that is the most critical part of the fortrabbit infra — while preparing and testing this update in our staging environment.

Posted 9 months ago. May 14, 2018 - 10:53 UTC

Issues with deployment in EU is now resolved.
Posted 9 months ago. May 14, 2018 - 00:18 UTC
There are still some issues which are identified, we are working to resolve it as soon as possible
Posted 9 months ago. May 13, 2018 - 23:45 UTC
Deployment issue resolved in EU region, We are still monitoring in case of any anomaly.
Posted 9 months ago. May 13, 2018 - 23:42 UTC
We are currently investigating deployment issues, probably related to scheduled maintenance earlier today.
Posted 9 months ago. May 13, 2018 - 20:20 UTC
This incident affected: Deployment EU.