Server architecture changes

Learn about the latest news here.
Post Reply
Marwolf
Level 80
Posts: 87
Joined: Mon Dec 28, 2020 4:17 pm
Location: Luna
Contact:

Server architecture changes

Post by Marwolf »

The primary Intel NUC server running VMware ESXi that was hosting the databases, website, and all game worlds except Uranium and Coleslaw which are on their own dedicated Intel NUC server ended up crashing yesterday when the datastore ran out of storage. It shutdown the virtual machine hosting everything to alert that it was out of space and would not let me add storage nor power on the VM to recover files. It would have taken far longer to copy 2TB of VM disk files off than to simply rebuild from scratch on another Intel NUC server a bare metal setup without any VMware ESXi virtualization complexity so after six hours and with Logg's offsite backup file restoration assistance, we managed to get 90% of everything back online last night.

The webclient and the wiki still need to be restored but both require some rearchitecting to work properly. A stopgap measure had been implemented to get them working for a while but it was not a good long term solution, similar to the VM running out of space. I had deleted an extra VM to free up space but eventually that ran out yesterday. There will be time tomorrow and this week to dig into a better design for hosting and disaster recovery / backups.

I want to deploy a replacement bare metal Intel NUC server for Uranium/Coleslaw matching the one I just did for the primary server hosting one. I'll automate database, PCAP, log, and file backups with a third intel NUC dedicated only to NAS backups and centralized security monitoring and then look into ways to simplify the Docker containers we use for services, simply the various shell scripts used during deployment and set up, and do a documentation overhaul from scratch as it has been about five years since the project began and many things have evolved over time from a production hosting perspective.

Thank you Logg for keeping archival backups of the database, files, logs, PCAPs, and everything else, you really saved the day!
PUREONE
Level 3
Posts: 2
Joined: Sun Feb 19, 2023 1:10 pm

Re: Server architecture changes

Post by PUREONE »

I still can't access the android client
PUREONE
Level 3
Posts: 2
Joined: Sun Feb 19, 2023 1:10 pm

Re: Server architecture changes

Post by PUREONE »

Just says unpacking
Kwharold305
Level 3
Posts: 1
Joined: Sun Feb 19, 2023 1:52 pm

Re: Server architecture changes

Post by Kwharold305 »

Mine says same thing never loads, my web client is not loading either.
Marwolf
Level 80
Posts: 87
Joined: Mon Dec 28, 2020 4:17 pm
Location: Luna
Contact:

Re: Server architecture changes

Post by Marwolf »

Gotcha. Sounds like we need to do some more android troubleshooting to see what’s going on.
whats good
Level 3
Posts: 3
Joined: Sun Jan 29, 2023 6:33 pm

Re: Server architecture changes

Post by whats good »

Would you guys happen to know when the OpenRSC wiki is back up?
Wine
Level 10
Posts: 12
Joined: Thu Jan 26, 2023 4:20 pm
Location: San Francisco, CA

Re: Server architecture changes

Post by Wine »

web client is working for me?
Post Reply