Deployment issue in EU

Resolved
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 14 hours and 32 minutes

There was a long downtime, yesterday evening for the deployment service (SSH/Git) for the whole EU area, around 4 hours, connected to a scheduled maintenance done yesterday. Luckily Sunday evening/night is not prime time when it comes to deployment — and it was only deployment, not web delivery — still that was more downtime, as we are comfortable with.

The technical issue was quite complex, as usual in such cases. A simple explanation goes like this: There was a complication when restarting the service after upgrade, resulting in an unstable state. We immediately cloned the service, but that took a little time and after that was finished, numerous other problems appeared.

We — of course — learned a lot and we are going to implement further steps in hardening the platform, by technical improvements and better procedures protocol.

We have probably focused too much on the web delivery part — as that is the most critical part of the fortrabbit infra — while preparing and testing this update in our staging environment.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 58 minutes

Issues with deployment in EU is now resolved.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 24 minutes

There are still some issues which are identified, we are working to resolve it as soon as possible

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 21 minutes

Deployment issue resolved in EU region, We are still monitoring in case of any anomaly.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

We are currently investigating deployment issues, probably related to scheduled maintenance earlier today.

Began at: