Internal updates maintenance EU

Known regressions and follow up issues:

• We have found and fixed another related issue where TLS certificates periodically have not been available
• The before mentioned Ghostscript library is missing on most Nodes, affects PDF manipulation with ImageMagick
• The GeoIp database is missing on most Nodes

We are now closing the maintenance window since all front facing services are upgraded. The way was a bit rocky. We apologize for the extended downtime. Thanks for your patience and understanding. Remaining is the deploy service which is planned for tonight. There is going to be an extra maintenance event.

We plan another maintenance in the near future to apply some minor fixes related to this maintenance.

We are expanding the window for the maintenance one more time, for two more days in Europe. We have faced some unexpected issues with the connectivity of the Memcache service (see dedicated incident). In the next days we proceed with the maintaining Pro Apps.

We are making progress with Pro Apps in EU. Some minor regressions became apparent. One is related to Ghostscript library, which is used by ImageMagick to work with PDF files. We plan to patch that soon. Please report issues if something is not working as before.

Starting with Pro Apps in EU today. This includes our client Dashboard as well.

It takes longer than expected, we are extending the maintenance window. The Universal Apps are mostly done. We will continue with Pro Apps.

Apart the described re-deploy issues, we have had some additional issues with SSH/SFTP connectivity and there are some other known minor issues left.

It's more rocky than we expected. Sorry for the inconvenience. Thanks for your understanding.

We have seen that a couple of re-deploy issues appeared with the Apps. This only affects a small number of Apps. Contact us if you think your App is in a broken state after the maintenance. We will try to proactivly mitigate such issues by restoring from a snapshot.

Re-deploy explanation

The maintenance triggered a redeploy of Git for your App. So all the latest changes from your Git have been re-applied to your App.

This actually is a feature to assure that everything is up-to-date. But in some cases this can cause trouble, especially when you have been mixing deployment methods. For instance, you have started using Git, but later on switched to use SSH or SFTP or you just have been applying updates from some web installer (update WordPress or Craft CMS from the control panel). Those changes might have been overwritten now.

Usually we ask not mix deployment methods in such ways. You can reset your remote Git repo here to make sure that will not happen again in the future when you plan to continue to work with SSH/SFTP for now. You can find a snapshot in our backups (when booked).

Here is more on the topic: help.fortrabbit.com/app#toc-version-mismatch-without-any-changes-made

This way you can make sure the sync process does not overwrite existing files in the future:

1. reset the remote repo > https://help.fortrabbit.com/git-deployment#toc-resetting-the-remote-repo
2. create a empty repository locally
3. commit something non-destructive (like a foo.txt in the root)
4. add your fortrabbit remote
5. push the changes to the main of the remote

It's fully non-destructive for your current "webspace". That will not fix the damage done now maybe but make sure it will not happen again.

We are making good progress a lot of Apps in EU have been moved already. Some clients are having issues due to changed IPs of their App. As a reminder, fortrabbit will not guarantee fixed IPs. https://help.fortrabbit.com/quirks#toc-outgoing-ip-address

The scheduled maintenance is now underway. We'll keep you updated on our progress.

We are going to run a maintenance sprint over 2 weeks. Within that time frame we will deploy internal software updates. We need to take each App hosted in EU down for a little while. We aim for less than 5 minutes individual downtime, but in certain cases it can take several hours until an App will become available again. Sorry, we can not say when which App will be taken down when, upfront. The maintenance will be carried out in the evening hours starting at 19:00 CEST spawning into the night. Thank you for your understanding.

https://blog.fortrabbit.com/september-2023-updates