Meltdown and Spectre patch

Duration: 4 days and 9 hours
Complete
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Thanks to all clients for understanding this special situation.

Sorry that we have not been able to plan it much better. We have finally spread the tasks over a few days. The total downtime was less than anticipated.

We have learned that some clients have used "Cowboy deployment" methods: mixing Git and SFTP. We are better prepared for that in the future.

Some hints suggested that the required patches will result in CPU performance loss (reportedly as much as 30 percent). So far, we are happy to say, that we don't see anything alike.

Let's go back to business.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

was completed long time ago.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

We are mostly done, some Memcache Nodes still have to be restarted/rebooted — scheduled for late evening hours (EU) today, early morning hours (US) tomorrow.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

We are in the progress of updating. Nodes and services are getting patched one by one, slowly, it can take a few days. we are monitoring for issues.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Scheduled maintenance is currently in progress. We will provide updates as necessary.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Maintenance will begin as scheduled in 60 minutes.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Sorry for the delay: Restart operations postponed (probably tomorrow), as the required kernel are still not in the available.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Postponed for Tuesday Wednesday evening (Berlin time) to make sure to have the correct Kernel patches included.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

Now after AWS has patched on their underlying level, we are preparing our systems against the recent Meltdown and Spectre exploits. It will be deployed as soon as possible, once the official Kernel patches are released, likely on the 10-01-2018. The update will cause a little downtime:

  • ~30 minutes for Universal Apps and Professional Apps not on production plans
  • < 3 minutes for Professional Apps on production plans
  • ~1 hour for deployment (Git/SSH/SFTP and MySQL tunnel)

We will update this page on that matter.

Began at: