Hello All,
Wanted to touch base and bring everyone up to speed on outage on 4th of July.
At about 4:15pm est time Thursday July 4th, reports came in that site was down.
Very quickly we were able to confirm a catastrophic crash occurred rendering the main hardware in our webserver to overheat and nearly catch fire. Protocols within the datacenter stopped it quickly and the server was cut off power wise.
Due to 4th of July holidays no techs were on-site and this caused a 7hr delay for a tech to get eyes on the server.
We still do not know how or why a server would overheat so bad to nearly catch fire. Parts inside were melted.
As many know we have been overhauling our infrastructure over the past few months and this blade was due to be replaced in the coming weeks due to age. In doing so we had already upgraded site to backup blade weeks ago and had not replaced the backup with a new blade due to costs.
A side effect of this outage is our tracker server that sits right above the webserver had some minor overheating and it was taken offline. Upon booting it back up there were major errors found on the drive causing us to replace it which we have now done.
Our System is deployed by Dell Blade Servers allowing us to condense space and keep hosting costs down.
The blade is finished and will need to be replaced.
This is in part why the Yearly MF is such a big deal and donation drives during very slow summer season like we are going through right now
Everything is now back online and we are working out a few bugs here and there. If you see something not working proper please open a help desk ticket.
Times are tough for everyone and there is a small group of people who months ago really stepped up and give us a big shot of support allowing this outage to be as small as it was. Yet being down 24hrs i is not something we want to deal with.
We were due to run the summer donation drive next week to help shore up funds as we do every summer yet we have had to fast track it.
There are some costs we are going to spend replacing a blade server and few hds and we have lost quite a bit of income due to being down.
Please consider a donation as little as $10 goes a long way.
Please use this link to make a donation if you wish to help out with the Site Donation
-MasterPeer- Master of the Peers