Site Down Time

[Oddity]

We had an unfortunate outage over the past few days that badly compromised our database. This appears to have begun on the 5th, but went unnoticed until late on the 8th/early 9th, then crashed entirely. There were sporadic issues during that time but nothing to portend what would come. The most recent clean backup we had available was on November 5th. That means the activity from approximately midnight on the 5th until it crashed was unrecoverable.

I'm really sorry, especially to everyone who made a post or a reply during that time.

On the bright side, we took advantage of the need to rebuild most of the database to upgrade the site to the latest version of vBulletin, so that is future planned downtime we have avoided, so that is nice.

A huge thanks to OldGrumpyDad, our Tech Volunteer, for all his effort putting us back together over the past couple days.

Thanks for your patience.

Cheers

[Bonnette]

You and GrumpyOldDad are the best! Thank you for making the best of the downtime by rebuilding the database and upgrading vBulletins.

[Oddity]

‘GrumpyDad is the tech genius making it all happen. I’m usually just along for the ride!

[juniorsenior]

Thank you for the notice! I was wondering what was going on

[Treeza04]

Thanks for all you techies do !

[pfcs49]

most grateful. you guys ROCK!

[Oddity]

OldGrumpyDad is the wizard. I’m just the communications (and part-time database) guy!

[ChesBay]

You guys are great. Thanks for all your excellent work!

[Jim]

Thank you both, we appreciate you!

[triumph]

Thanks for your work!

[Domosoyo]

Thank you!

[Oddity]

Unfortunately we aren’t out of the woods yet. Upon a further detailed review, there is failed DIMM chip on the main hardware motherboard, and the new hardware that arrived for the RAID array is incompatible. When it rains it pours. The bad DIMM was likely the beginning of our issues and database corruption that followed. The system is stable, for the time being, bad parts isolated and out of the equation, but some more previously unscheduled downtime, for further repairs, is back on the table. Perhaps as soon as later today. I’ll keep this thread updated as I can. Thanks.

[Bonnette]

I'm in awe of your ability to track down and fix these technological problems. Thank you so much.

[Oddity]

I am too! The depth of knowledge ‘Grumpy brings to the table is amazing. He lives right near our data center (I’m ~3 hours away), so he’s able to get hands on with our servers and really figure things out. Remote admin only goes so far! Cheers and thanks for the encouragement.