Website delivery issues for a small number of Apps

Post mortem

On Sunday morning, while trying to understand a client support case, we discovered that a whole Universal App Node was unresponsive, affecting web delivery for small number of Apps in EU. The client already reported the issue 9 hours earlier (2nd of March, 20:00 UTC), unfortunately just outside our support hours.

The issue turned out to be a DDOS attack targeting a specific App on that Node. We immediately started mitigation steps. After a while the flood of requests slowed down. We kept monitoring the situation across the whole Sunday.

A DDoS is considered force majeure. Some of our clients are using CloudFlare for DDoS protection in addition.

We will improve monitoring and alerting to better catch such issues on infra level and also update existing mitigation routines.

We are still monitoring the situation. Operations should be stable since 09:30 UTC today. We plan to close this incident on Monday.

The attack is ongoing. We have take one website offline completely, as it seems to be the target of the attack. We are also still block a great number of requests. So far the situation for clients should be mostly stable.

We are now monitoring the situation and will continue to do so for another while in which the situation may become worse again.

Mitigations are in place now and the situation should be OK for most clients. We are still looking into it. It might become worse again.

This is a more sophisticated DDOS attack, targeting the IP directly from a wide range of sources. We are working on mitigations.

Since Saturday, around 20:00 UTC a couple of Apps can not be reached from the internet. The request times out before reaching the PHP processes. As far as we can see, this seems to be a DDoS attack on one of the Apps. We are looking into mitigations.