Service interruption

Resolved
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 28 minutes

This partial service interruption was caused by a failure of one of our storage nodes. It suffered from a temporary network disconnect which was too short for the failover to take place. We will once again review our failover mechanisms to better compensate such conditions in the future.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 23 minutes

Everything should be back to normal again now. Pingdom says about 5 minutes downtime which reflects our measurement.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 7 minutes

We have identified the issue. It is a storage issue (one storage node), still investigating. The failover did not worked as expected.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

We see "maintenance" messages on some sites/Apps. We are looking into this right now.

Began at: