On Sunday, 17th of Feb, 20:38 UTC, a Web Delivery Node in US went down. This very Node was cause of problems in the recent days.
So instead of a restart we decided to replace the hardware. After that, all monitoring channels responded with 200 OK and some Apps also seemed to be OK by manually checking. So we concluded that the issue would be fully resolved.
But, we only found out in the morning (Berlin time) that public IP of the Node changed. That was not anticipated. We re-connected the IPs and only then the Apps came back up.
All services where fully restored at around 08:36 UTC.
We are very very sorry for the inconvenience this has caused. It was a long downtime and it was the not the first on this Node in recent days. Fortunately only a small number of clients were affected. The Node is now replaced. We are confident that all issues are finally resolved.
Posted about 1 month ago. Feb 17, 2019 - 11:55 UTC