Server Upgrade February 2023

On February 14th 2023 I started upgrading our server infrastructure to handle the recently increased load.

First step of the process is to pull a full backup of the server in case we need to recover from data loss.

At around 12:00 CET I started the full backup. Due to high load this process took far longer than expected. Intermittently I had to shut down the web server to decrease load so the backup could finish more quickly and to allow the background jobs to finish running instead of piling up.

I'll update this post as I go through the upgrade process.

At 13:50 CET, the backup finished. I restarted the web server to bring the instance back online and let it sync up to the rest of the Fediverse. Server upgrade will commence shortly.

At 14:15 CET I triggered the server VM upgrade with our hosting provider.

At 16:24 CET our hosting provider has initiated the upgrade on their end.

At around 17:00 CET the server was rebooted automatically and was available again after about 6 minutes with the new upgraded configuration.

The upgrade is concluded and the Mastodon instance returns to normal operation.

Last update: 2023-02-14 17:40 CET