Server architecture changes
Posted: Sun Feb 19, 2023 11:25 am
The primary Intel NUC server running VMware ESXi that was hosting the databases, website, and all game worlds except Uranium and Coleslaw which are on their own dedicated Intel NUC server ended up crashing yesterday when the datastore ran out of storage. It shutdown the virtual machine hosting everything to alert that it was out of space and would not let me add storage nor power on the VM to recover files. It would have taken far longer to copy 2TB of VM disk files off than to simply rebuild from scratch on another Intel NUC server a bare metal setup without any VMware ESXi virtualization complexity so after six hours and with Logg's offsite backup file restoration assistance, we managed to get 90% of everything back online last night.
The webclient and the wiki still need to be restored but both require some rearchitecting to work properly. A stopgap measure had been implemented to get them working for a while but it was not a good long term solution, similar to the VM running out of space. I had deleted an extra VM to free up space but eventually that ran out yesterday. There will be time tomorrow and this week to dig into a better design for hosting and disaster recovery / backups.
I want to deploy a replacement bare metal Intel NUC server for Uranium/Coleslaw matching the one I just did for the primary server hosting one. I'll automate database, PCAP, log, and file backups with a third intel NUC dedicated only to NAS backups and centralized security monitoring and then look into ways to simplify the Docker containers we use for services, simply the various shell scripts used during deployment and set up, and do a documentation overhaul from scratch as it has been about five years since the project began and many things have evolved over time from a production hosting perspective.
Thank you Logg for keeping archival backups of the database, files, logs, PCAPs, and everything else, you really saved the day!
The webclient and the wiki still need to be restored but both require some rearchitecting to work properly. A stopgap measure had been implemented to get them working for a while but it was not a good long term solution, similar to the VM running out of space. I had deleted an extra VM to free up space but eventually that ran out yesterday. There will be time tomorrow and this week to dig into a better design for hosting and disaster recovery / backups.
I want to deploy a replacement bare metal Intel NUC server for Uranium/Coleslaw matching the one I just did for the primary server hosting one. I'll automate database, PCAP, log, and file backups with a third intel NUC dedicated only to NAS backups and centralized security monitoring and then look into ways to simplify the Docker containers we use for services, simply the various shell scripts used during deployment and set up, and do a documentation overhaul from scratch as it has been about five years since the project began and many things have evolved over time from a production hosting perspective.
Thank you Logg for keeping archival backups of the database, files, logs, PCAPs, and everything else, you really saved the day!