To All Members Of The Wood Works

Martin and I spoke again this morning about our situation and are in complete agreement on the fix. Martin prepared in words what has taken place and asked me to post it here on Sawdust and Cowpies. Here tis...

*****************************************************

On Monday we experienced a catastrophic hardware failure on my server. Not only was the system damaged beyond repair, but it also wrote something to thedisk array that somehow managed to overwrite the partitioning information on the drives. To make life even more 'interesting', the tape that I was using for backups decided to self destruct in the drive.

I have spent the last 4 days pulling together the parts for a new web server and attempting to recover the data on the drives. On Wednesday, I decided that the problem with the drives was beyond my experience to repair and I engaged ESS Data Recovery to assess the drives and attempt a recovery. The initial prognosis was a 70% or less chance of recovery of the data. Return time on the attempt is approximately 1-2 weeks. Unfortunately, upgrading this to a 24X7 recovery attempt was well beyond what I can afford to pay for the data. As it stands now, if they are able to recover the data, there will be a 1K$ charge. There is no charge if the data is not recoverable.

Jace, the other moderators, and I have been discussing the options throughout the week. I think the time has come to relaunch the sites 'from scratch' so that we can begin rebuilding the communities. There is still a chance that within the next week or two that the original data might be recovered, but as the prognosis was not promising, I believe it is time to roll up my sleeves and get things going again. If ESS is able to recover the data, I have already been researching methods for reimporting the old posts back into the new forums. If this happens, I will attempt to do this in a non-disruptive manner. I do not however want to delay bringing the sites back up while we wait for a recovery that might not happen.

This has been a very humbling experience for me. I pride myself in my ability to manage IT operations. Coincidentally, we had a few other hardware issues in the data center that happened in the same time frame. That partly slowed my efforts on The Wood Works and other sites as those servers are part of the job that pays my bills. Luckily, all of those systems are functional. I'm not sure at this point if the damage was earthquake related, or if other factors were present. Either way, it took 3 points of failure (Bad server hardware, mirrored disk array failure, backup tape failure) to cause the data loss we experienced. For this I offer my most sincere apologies. I understand fully that the value of a forum is not the server or the software, but all the time and effort that are put into the posts contained within by the members.

I have put additional procedures in place to reduce my reliance on tape backups. While I can not say that we will never experience another issue in the future, I can guarantee that this situtation will not occur again. Future backups will be transfered offsite and will not be reliant on tape technology.

The goal right now is the get everything back in place for a Sunday relaunch. Nicky and I will be rebuilding the templates and configurations tonight and tomorrow to make this happen. Unfortunately, everyone will need to create new accounts. If possible, please retain your original account name, as this may help in the event that we are able to migrate the old data back in place. It may take a few more days for all features to be funtional again, specifically member gallerys and banner add rotation.

I appreciate the words of support that Jace has received and relayed to me from all of you. I hope to see everyone back together in the forums very soon.

-Martin

**********************************************************

So, boys and girls... that's where we're at. If you have any questions, please by all means email me at webers@cvalleyDOTnet I'll do my damnedest to get the proper answer back to you.

I apologize for this happening as well. Thank You for all your understanding and support.


Best Regards,
Jace