Resolved

We've now resolved the incident. Thanks for your patience.

There was an issue with our Object Storage Component in EU that started 09:20 UTC and was resolved 10:35 UTC. The Object Storage was unresponsive or slow.

Usually this shall only have a small effect since most static requests should be cached. Most clients will have not noticed this at all for web delivery from the Object Storage, since usually images and other files are just hot-linked from the website.

It seems that some clients are piping requests to the Object Storage over PHP in a blocking manner. These requests failed and blocked whole websites. We recommend to review if such piping is required and if so if it is possible to make it fail gracefully.

Avatar for
Recovering

We think to have fixed the issue and are now monitoring the situation.

Investigating

We are currently looking into problems with loading websites. You might see 504 time out error or have issues with the Object Storage. This applies to some Pro Apps in EU only.

Avatar for
Began at:

Affected components
  • EU
    • Pro Apps
      • Memcache
      • Object Storage
      • Web delivery
      • Worker
    • Universal Apps
    • MySQL
    • Deployment
    • Backups