Resolved

One of the our memcached nodes suffered a local DNS issue. Since the node itself was responsive from the outside, it took us about 20 minutes to identify the issue. Once identified, we were able to solve it and the memcache node is now up and working again.

We're sorry for any inconvenience and are now investigating why the local DNS issue was not properly detected by our monitoring and will subsequently improve our auto-healing so it can cope with this kind of service failure properly.

Affected were all Apps using that memcache node as single instance and Apps using multiple memcache instances, but were not configured for a redundant memcache setup. You can find our recommendation for a redundant memcache setup here: https://help.fortrabbit.com/memcache-pro#toc-redundant-setup

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Began at: