Incidents reported on status page for fortrabbit https://status.fortrabbit.com/ https://d1lppblt9t2x15.cloudfront.net/logos/7e49e705aabf78be7cbf4fb209a53961.png https://status.fortrabbit.com/ en https://status.fortrabbit.com/incident/907887 Fri, 29 May 2026 11:16:00 -0000 https://status.fortrabbit.com/incident/907887#a8bae81fc65aa0224e614840adeda3c482711c620f7ec9a45d3bdf8096406c2f Post mortem Universal Apps run shared host systems. These host systems have - of course - limited file storage. Initially we enforced hard file system quotas, but an underlying bug in BTRFS caused issues with that. So we lifted the hard limits and allowed exceeding the including file system limits. For Universal Apps, there is no automatic scaling, so that solution is actually better for clients too. We overbook storage, roughly by the factor of three, which would allow clients to use as twice as storage as the largest Uni plan. We regularly check for Apps that are exceeding the limits and write mails. Lately we have to do that more often. Most commonly, excessive file storage usage is attributed to code/config misconfiguration. Most often find a lot of old local backups files (maybe some plugin?), but also a lot of cache files. Specifically the later is getting an issue more and more. Maybe because of bots more aggressivly crawling websites, that offer an unlimited number of possible pages. We have health checks in place, alerting us on storage issues. In this case, the storage filled quickly, causing all kind of random errors. We pro-actively removed cache folders, which caused higher load on the host system, when Apps refilled the caches. This architecture is of course not prefect. It's one of the many reasons we decided to create a new platform from scratch. The new platform solves this on multiple levels: - Larger storage plans - Optional "Autoscaling": either capped, or upgrade After monitoring, we consider the issue now resolved. https://status.fortrabbit.com/incident/907887 Fri, 29 May 2026 07:52:00 -0000 https://status.fortrabbit.com/incident/907887#ac9e4b957d6e8f0ba89d99fe0cb8b158e52f57fd409ce02d55c1b73b0f92a2ae Apps should be back for now. But we are still investigating. https://status.fortrabbit.com/incident/907887 Fri, 29 May 2026 07:20:00 -0000 https://status.fortrabbit.com/incident/907887#f47d461abf24da4f1b70a4a20f3bdd1b96383677300e295d3363f016edfacfe8 We are looking into web delivery issues for about 30 Apps hosted in the EU region of the old platform. The issues may have started around 06:00 UTC. Not all Apps are down, just some, they return random errors or timeouts. https://status.fortrabbit.com/incident/890209 Tue, 12 May 2026 05:39:08 -0000 https://status.fortrabbit.com/incident/890209#d6cf5f451cd2abb403a832d89b0eded46a30c5f303b30fcd923be85d9392d8f0 Maintenance completed https://status.fortrabbit.com/incident/890209 Tue, 12 May 2026 05:00:08 -0000 https://status.fortrabbit.com/incident/890209#b0195ecfea0f2cd355144682a0145e330034f4796119f265ed8bd4b53719266b We will roll out some security patches during the maintenance window. This will not affect customer facing software version changes. The expected downtime per app is under a minute. We run this in batches for both regions US and EU. BTW: We have migrated our status tool and cleaned up a bit. So this notification may look different now.