All systems are go

Don't agree with this? Please let us know

  • US

    United States data center location

    • Pro Apps

      US

      • Web delivery

        US Pro

      • Object Storage

        US Pro

      • Memcache

        US Pro

      • Worker

        US Pro

    • Universal Apps

      US web delivery

    • MySQL

      US Universal and Pro Apps

    • Deployment

      US Git, SSH, SFTP for Pro and Universal

  • EU

    European data center location

    • Pro Apps

      EU

      • Memcache

        EU Pro

      • Object Storage

        EU Pro

      • Web delivery

        EU Pro

      • Worker

        EU Pro

    • Universal Apps

      EU web delivery

    • MySQL

      EU Universal and Pro Apps

    • Deployment

      Git, SSH, SFTP

  • Dashboard
Previous Incidents

[Resolved] Rebooting Universal node

This incident lasted 14 minutes.
  • Universal Apps
Sun, 16 Jun 2019
14:55:31 UTC

Re-Initializing in progress

15:09:45 UTC

We've now resolved the incident. Thanks for your patience.

[Resolved] Network connectivity issues with one Node (EU)

This incident lasted 14 minutes.
  • Universal Apps
Thu, 13 Jun 2019
08:20:24 UTC

We are replacing this node.

08:34:36 UTC

We've now resolved the incident. Thanks for your patience.

[Complete] Under the hood maintenance, part 4 — EU (+ US)

This was a planned incident, it took 4 hours and 2 seconds.
  • Deployment
  • Deployment
  • Memcache
  • Universal Apps
  • Universal Apps
  • Web delivery
Wed, 12 Jun 2019
12:00:11 UTC
Overview

We have to run another update round. This time we expect very little or not downtimes as we are mostly updating background services.

The redirect service will have a glitch (minutes) in EU. (-> naked to www forward)

The tunnel service for connecting to MySQL from local will also have a short downtime in US and EU.

  • Scheduled for Wed, 12 Jun 2019 16:00 UTC
  • Expected to take 4 hours
16:00:00 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

20:00:01 UTC

The maintenance is now complete. Thanks for your patience.

[Resolved] Web delivery issues

This incident lasted 2 hours and 9 minutes.
  • Universal Apps
Tue, 11 Jun 2019
10:39:43 UTC

We are seeing some issues with web delivery for a few clients on Universal Apps in EU.

11:59:54 UTC

We are still looking into that. This also affects SSH/SFTP connections — only a small number of clients, in EU, only Universal Apps.

12:49:26 UTC

We've now resolved the incident. Thanks for your patience.

[Resolved] Deploy and SSH issues in EU

This incident lasted 1 hour and 38 minutes.
  • Deployment
Tue, 11 Jun 2019
10:20:38 UTC

We are currently experiencing deployment and SSH connectivity issues in EU region.

11:59:10 UTC

We've now resolved the incident. Thanks for your patience.

[Complete] Under the hood updates, part 3 — EU (reposted)

This was a planned incident, it took 4 hours and 3 seconds.
  • Dashboard
Mon, 10 Jun 2019
10:43:14 UTC
Overview

This is hopefully the last part of our maintenance marathon in EU. We plan to patch the last remaining Nodes. So again, this might affect web delivery of some of your Apps (if not done before or App is Pro with production plan). We plan not more than 30 minutes per App. Updates will be run in batches.

Reposted as we did a mistake in updating the wrong event.

  • Scheduled for Tue, 11 Jun 2019 18:00 UTC
  • Expected to take 4 hours
Tue, 11 Jun 2019
18:00:01 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

20:03:06 UTC

We are now starting an upgrade of some internal services. Our Dashboard might be unresponsive for some minutes during the upgrade.

22:00:03 UTC

The maintenance is now complete. Thanks for your patience.

[Complete] Under the hood maintenance, part 4 — US

This was a planned incident, it took 13 hours and 1 second.
  • Pro Apps
  • Memcache
  • Object Storage
  • Web delivery
Thu, 6 Jun 2019
16:13:24 UTC
Overview

So, progress is still slow but steady. We need to extend the maintenance window fo US clients one more day. The Universal Apps are done. Now we are about to update the Pro Apps. Pro Apps on production plans are likely to have near-zero downtime, others up to 30 minutes. It's running in batches. We also gonna update some background services which are not supposed to affect clients.

  • Scheduled for Fri, 7 Jun 2019 08:00 UTC
  • Expected to take 13 hours
Fri, 7 Jun 2019
08:00:02 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

13:22:12 UTC

So in case you are wondering, yes we are on it everything is running as expected currently.

17:21:13 UTC

We are still in process of upgrade, Professional and High available apps will stay online but may face slight hiccups as upgrade proceed.

We will update you as soon as we have more updates.

18:47:47 UTC

We are near completion of the upgrade in US region, it took more than expected time.

Fortunately not much disturbance is expected, if you see any persistent issues please contact support.

20:24:41 UTC

As we are finalizing updates we will test and make sure your apps are on updated nodes and ensure proper web delivery.

If your problem persists please contact support.

21:00:02 UTC

The maintenance is now complete. Thanks for your patience.

[Complete] Under the hood updates, part 3 — EU

This was a planned incident, it took less than 1 second.
  • Web delivery
  • Object Storage
  • Universal Apps
  • Memcache
  • Pro Apps
  • MySQL
  • Deployment
  • EU
Thu, 6 Jun 2019
14:08:23 UTC
Overview

This is hopefully the last part of our maintenance marathon in EU. We plan to patch the last remaining Nodes. So again, this might affect web delivery of some of your Apps (if not done before or App is Pro with production plan). We plan not more than 30 minutes per App. Updates will be run in batches.

  • Scheduled for Tue, 11 Jun 2019 18:00 UTC
  • Expected to take 15 minutes
14:11:16 UTC

We have noticed that some clients have used the IP of their App to route their domains. The IP of your App will change with this update and that we do not advice to use an IP for DNS routing, use the host name (App URL) as a CNAME record. This applies to "www" and any other sub domain records, not the naked domain (for which we are offering the redirect service, that IP stays of course).

Sun, 9 Jun 2019
16:14:44 UTC

Everything went smoothly so far. Deployment still disabled.

17:18:29 UTC

We've made a change to the schedule.

Mon, 10 Jun 2019
01:00:17 UTC

The maintenance is now complete. Thanks for your patience.

[Complete] Under the hood updates, part 2 — EU

This was a planned incident, it took 9 hours and 24 minutes.
  • Universal Apps
Thu, 6 Jun 2019
13:36:51 UTC
Overview

This is the planned second part of our big update marathon in EU. We expect this one to be mostly about web delivery. Expected downtime is 30 minutes max per App, excluding Apps that already have been updated in part 1 and Pro Apps on production plans (HA).

  • Scheduled for Mon, 10 Jun 2019 08:00 UTC
  • Expected to take 10 hours
Mon, 10 Jun 2019
08:00:04 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

09:47:06 UTC

We have started to update Universal Apps, expected is a downtime of around 25 minutes per App, is happening in batches.

16:49:43 UTC

We are on the last stretch here of our updates, two servers are causing us some issues so some sites has been down for longer than expected. Should be up soon!

17:24:40 UTC

Finally we are finished for today. All servers hosting Universal Apps in EU have now been upgraded. Please let us know in support if you have any further issues.

[Complete] Under the hood maintenance, part 3 — US (reposted)

This was a planned incident, it took 9 hours and 3 seconds.
  • MySQL
  • Pro Apps
  • Universal Apps
  • Web delivery
  • Object Storage
  • Memcache
Wed, 5 Jun 2019
18:23:08 UTC
Overview

Sorry, a REPOST here: We have accidentally posted this on the wrong day. It's supposed to run on Thursday, the 6th of June, of course:

So it takes much longer than anticipated, but - slowly - we make progress. We have to continue the progress of updating Nodes tomorrow. It's the same thing that has been happening today. The updates will include some downtime for web delivery in US, individual on a per App basis. We try to update this thread frequently.

  • Scheduled for Thu, 6 Jun 2019 08:00 UTC
  • Expected to take 9 hours
Thu, 6 Jun 2019
08:00:01 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

08:48:14 UTC

We are now proceeding with todays maintenance. Expected downtime for Universal Apps in US are around 30 minutes per App.

14:14:14 UTC

We have noticed that some clients have used the IP of their App to route their domains. The IP of your App will change with this update and that we do not advice to use an IP for DNS routing, use the host name (App URL) as a CNAME record. This applies to "www" and any other sub domain records, not the naked domain (for which we are offering the redirect service, that IP stays of course).

17:00:04 UTC

The maintenance is now complete. Thanks for your patience.

[Resolved] Deployment issues in US

This incident lasted 2 hours and 49 minutes.
  • Deployment
Wed, 5 Jun 2019
04:26:07 UTC

We are continuing to have issues in US Deployments, we are currently investigating the cause of this issue.

05:33:11 UTC

We are still on it.

07:15:31 UTC

We've now resolved the incident. Thanks for your patience.

07:18:18 UTC

The recent deployment issues in US are resolved for now. Deployment is working. You see, we have a series of problems with deployment since the updates on Monday. I have to admit that we are not fully confident to have solved the root issue now. We are still on it. — ƒrank

[Resolved] Deployment problems in US

This incident lasted 2 hours and 49 minutes.
  • Deployment
Tue, 4 Jun 2019
18:43:45 UTC

Sorry, we again see issues with deployment in US, currently only Git affected.

19:17:13 UTC

We are working on it. We also took SSH and SFTP down.

19:27:30 UTC

We've fixed the core issue, and are waiting for things to recover.

19:27:51 UTC

We are still monitoring the situation.

19:31:57 UTC

We believe there may be an issue, we're looking into it.

19:39:26 UTC

ARGH. Still not coming back. We need to dig deeper here.

20:23:04 UTC

We've fixed the core issue, and are waiting for things to recover.

21:33:24 UTC

Deployment service is healthy again.

[Complete] Under the hood maintenance, part 2 — US

This was a planned incident, it took 9 hours and 18 minutes.
  • Universal Apps
  • Web delivery
  • Object Storage
  • Memcache
  • Pro Apps
Tue, 4 Jun 2019
16:52:14 UTC
Overview

So after a shaky original under the hood maintenance, we still need to do the missing parts. We will run updates on the web delivery services in the US. Expected downtime for individual Apps is about 30 minutes. It will happen one by one. Pro Apps on High Availability (production) plans are expected to have minimal downtime.

More on the updates:
https://blog.fortrabbit.com/under-the-hood-updates-2019-06

  • Scheduled for Wed, 5 Jun 2019 08:00 UTC
  • Expected to take 10 hours
Wed, 5 Jun 2019
08:00:01 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

08:24:00 UTC

We are on it. Sorry for the inconvenience and downtimes on deployment during the last days one more time. We will provide a post mortem as soon as we have the time and all the details. Please be assured that we are on it — full power. We are currently still discussing issues and status and plan to proceed with the planned maintenance soon. We will update here.

15:29:12 UTC

We are still working and upgrading server by server, it is taking more time than expected. Downtime for single Apps should still be minimal.

17:18:27 UTC

We are finishing up for the day and will continue tomorrow.

[Complete] Under the hood updates, part 1 — EU

This was a planned incident, it took 17 hours and 13 minutes.
  • Deployment
  • Pro Apps
  • Object Storage
  • Memcache
  • Universal Apps
  • MySQL
  • EU
  • Web delivery
Tue, 28 May 2019
19:13:12 UTC
Overview

A bigger planned maintenance with many internal updates, lot's of minor and patch version changes. More details here: https://blog.fortrabbit.com/under-the-hood-updates-2019-06

  • Scheduled for Sun, 9 Jun 2019 08:00 UTC
  • Expected to take 15 hours
Thu, 6 Jun 2019
13:27:13 UTC

We have updated title and timing a bit here, based on current experiences with the same updates in our US region. We spliced the big maintenance in three parts now. On this first part we will mainly focus on deployment and therefore Git, SSH and SFTP will not work for a couple of hours in the EU. But there likely will also gonna be short downtimes for Apps as well. We expect up to 30 minutes for Universal Apps and Pro Apps on non-production plans (non-HA).

Sun, 9 Jun 2019
08:00:03 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

17:44:28 UTC

All the Pro apps are successfully migrated into new servers, Deploy is taking longer than expected, Good news is that it is in final stages,

we will post as soon as we have more updates.

if you face any issue with web delivery, please contact support.

19:27:01 UTC

We are still in process of upgrading deploy server, we are facing some issues which we are trying to resolve, we will update you as we have some.

20:29:15 UTC

Deploy server needs more time to restore, If there is any issue other than deployments or ssh please contact support.

22:48:00 UTC

We are still working on the upgrade, we will post updates as soon as we have some.

Mon, 10 Jun 2019
01:13:33 UTC

The maintenance is now complete. Thanks for your patience.

[Complete] Under the hood updates — US

This was a planned incident, it took 16 hours and 59 minutes.
  • Deployment
Tue, 28 May 2019
19:11:05 UTC
Overview

A bigger planned maintenance with many internal updates, lot's of minor and patch version changes. More details here: https://blog.fortrabbit.com/under-the-hood-updates-2019-06

  • Scheduled for Mon, 3 Jun 2019 08:00 UTC
  • Expected to take 17 hours
Mon, 3 Jun 2019
08:00:03 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

09:21:38 UTC

We started the roll out. Currently all actions performed in the Dashboard affecting US Apps will be delayed. Soon the deploy services will go down.

10:35:40 UTC

Now deployment is going down for US. So there is no Git, no SSH and no SFTP for a while. We don't know how long it will take, but we expect this to take a couple of hours. Apps will come back one by one.

14:05:39 UTC

So — it's taking longer than originally anticipated. Deployment services are slowly booting up, App by App now. We expect deployment to fully work again for everyone in a couple of hours.

15:00:03 UTC

The maintenance is now complete. Thanks for your patience.

15:02:10 UTC

The scheduled maintenance is now underway. We'll keep you updated on our progress.

15:05:19 UTC

Sorry, the deployment is not fully back up. It takes longer than anticipated currently. Everything except deployment should work now. Only deployment is not working in the US for some Apps. We will post updates....

18:49:30 UTC

We are still on it. Sorry that this is taking much longer than expected. We will post some updates on this as soon as we can.

Tue, 4 Jun 2019
01:00:01 UTC

The maintenance is now complete. Thanks for your patience.

[Resolved] Deployment issues in US

This incident lasted 1 week, 4 days, 17 hours, and 32 minutes.
  • Deployment
Thu, 23 May 2019
20:13:10 UTC

We are receiving reports of deployment related issues in US region, we are investigating the cause behind it.

20:33:09 UTC

We have identified the issue with the deploy server, we are working on the fix.

22:08:26 UTC

We have identified the issue and applied the fix, it may take some more time for the server to completely restore to its original state, Intermittent deployment issues are expected during the restoration process.

We will update as soon as restore is complete.

Fri, 24 May 2019
00:30:27 UTC

Restoration is complete, We can confirm that deployment should work without any issue, In case of any trouble please contact support.

Tue, 4 Jun 2019
08:59:32 UTC

We were forced to reset the deployment server again, it may take some time for the server to completely restore to its original state, Intermittent deployment issues are expected during the restoration process.

11:59:00 UTC

Still ongoing. Sorry for the inconvenience. We are on it.

13:46:01 UTC

Restoration is complete, We can confirm that deployment should work without any issue, In case of any trouble please contact support.

No further notices from the past 30 days.