Deployment issue in EU
Incident Report for fortrabbit
Postmortem

There was a long downtime, yesterday evening for the deployment service (SSH/Git) for the whole EU area, around 4 hours, connected to a scheduled maintenance done yesterday. Luckily Sunday evening/night is not prime time when it comes to deployment — and it was only deployment, not web delivery — still that was more downtime, as we are comfortable with.

The technical issue was quite complex, as usual in such cases. A simple explanation goes like this: There was a complication when restarting the service after upgrade, resulting in an unstable state. We immediately cloned the service, but that took a little time and after that was finished, numerous other problems appeared.

We — of course — learned a lot and we are going to implement further steps in hardening the platform, by technical improvements and better procedures protocol.

We have probably focused too much on the web delivery part — as that is the most critical part of the fortrabbit infra — while preparing and testing this update in our staging environment.

Posted 12 months ago. May 14, 2018 - 10:53 UTC

Resolved
Issues with deployment in EU is now resolved.
Posted 12 months ago. May 14, 2018 - 00:18 UTC
Identified
There are still some issues which are identified, we are working to resolve it as soon as possible
Posted 12 months ago. May 13, 2018 - 23:45 UTC
Monitoring
Deployment issue resolved in EU region, We are still monitoring in case of any anomaly.
Posted 12 months ago. May 13, 2018 - 23:42 UTC
Investigating
We are currently investigating deployment issues, probably related to scheduled maintenance earlier today.
Posted 12 months ago. May 13, 2018 - 20:20 UTC
This incident affected: EU (Deployment EU).