Today, Thursday 8 September, we will perform major software maintenance on the Fuga platform.
We will be upgrading the Fuga platform to the OpenStack Mitaka release. This concerns all components, including those related to networking, the dashboard and the APIs.
The maintenance will start at 07:00 CEST and will be ongoing throughout a large part of the day.
The timetable is as follows:
Phase 1. 07:00 – 08:00 CEST – Storage upgrade
Phase 2. 08:00 – 12:00 CEST – Management/controller layer upgrade
Phase 3. 12:00 – 18:00 CEST – Compute nodes upgrade
Below, you can find a live registration of the upgrading process. In case you prefer Twitter to follow the upgrading process, visit https://twitter.com/FugaCloud
07:03 CEST - Start upgrade
We've started the upgrade to Mitaka. First up: upgrade the storage (Ceph) nodes.
07:15 CEST - Ceph controllers upgraded
The Ceph controllers have all been upgraded. Starting on the storage itself now.
07:50 CEST - Storage upgrade running, preparing next step
The storage upgrade is going steady, but taking a bit longer than expected. We're setting up to start phase 2 in parallel.
08:15 CEST - Continuing storage upgrade in the background
Applying the changes to the Ceph storage takes far more time than expected, but has no impact on availabilty or performance. We're still running that in parallel to the rest of the upgrade.
08:35 CEST - Object storage problems
Although Ceph storage is still available, we do however experience issues with the object store. We're working to resolve this with the highest priority.
08:45 CEST - Databases successfully upgraded
As part of phase 2, all databases have been successfully upgraded.
09:25 CEST - Update: object store problems
It seems like the problem with the object store can only be fixed once all storage has been upgraded. We're starting on the last node right now.
10:20 CEST - Storage upgrade completed. Object store still has problems
Although the update to the storage systems has been completed, the object store is still not online. We're looking very hard into the cause. The update to the management nodes is ongoing.
10:50 CEST - Updating controller nodes
At the moment, the controller nodes are being updated. The APIs and Dashboard are therefore unavailable to the moment. Instances remain online.
12:10 CEST - Almost ready to test new controllers
We're finalizing the last changes to the controller nodes. Once this has successfully been tested, we'll gradually move functionality to the new nodes. This may cause short disruptions in connectivity to your instances.
14:45 CEST - Object store available, two controller upgrades completed
We've managed to restore the object store functionality, although listings may be unavailable from time to time while we're still working on the platform. Two controller upgrades have been completed at this time.
15:05 CEST - Rerouting networks
We're now rerouting networks and activating new configurations. Expect some downtime for your instances.
17:00 CEST - End of maintenance window, compute node upgrade postponed
All management nodes have now been upgraded to Mitaka. Instances, services, networks, the APIs and the Dashboard are all available once more. We’re currently double checking all functionality. The object store is still showing some hiccups, so we’re still looking into that.
Due to the delay we experienced during phase 1 and 2 of the upgrade process, we will postpone the upgrade to the compute nodes to a later date. We will inform you about a new upgrade path as soon as possible. Delaying this part of the maintenance does not have an effect on the platform or your instances.
Tomorrow, we’ll send a full report of today’s work to all Fuga users.