Friday, April 25, 2008

To All Members Of The Wood Works

Martin and I spoke again this morning about our situation and are in complete agreement on the fix. Martin prepared in words what has taken place and asked me to post it here on Sawdust and Cowpies. Here tis...

*****************************************************

On Monday we experienced a catastrophic hardware failure on my server. Not only was the system damaged beyond repair, but it also wrote something to thedisk array that somehow managed to overwrite the partitioning information on the drives. To make life even more 'interesting', the tape that I was using for backups decided to self destruct in the drive.

I have spent the last 4 days pulling together the parts for a new web server and attempting to recover the data on the drives. On Wednesday, I decided that the problem with the drives was beyond my experience to repair and I engaged ESS Data Recovery to assess the drives and attempt a recovery. The initial prognosis was a 70% or less chance of recovery of the data. Return time on the attempt is approximately 1-2 weeks. Unfortunately, upgrading this to a 24X7 recovery attempt was well beyond what I can afford to pay for the data. As it stands now, if they are able to recover the data, there will be a 1K$ charge. There is no charge if the data is not recoverable.

Jace, the other moderators, and I have been discussing the options throughout the week. I think the time has come to relaunch the sites 'from scratch' so that we can begin rebuilding the communities. There is still a chance that within the next week or two that the original data might be recovered, but as the prognosis was not promising, I believe it is time to roll up my sleeves and get things going again. If ESS is able to recover the data, I have already been researching methods for reimporting the old posts back into the new forums. If this happens, I will attempt to do this in a non-disruptive manner. I do not however want to delay bringing the sites back up while we wait for a recovery that might not happen.

This has been a very humbling experience for me. I pride myself in my ability to manage IT operations. Coincidentally, we had a few other hardware issues in the data center that happened in the same time frame. That partly slowed my efforts on The Wood Works and other sites as those servers are part of the job that pays my bills. Luckily, all of those systems are functional. I'm not sure at this point if the damage was earthquake related, or if other factors were present. Either way, it took 3 points of failure (Bad server hardware, mirrored disk array failure, backup tape failure) to cause the data loss we experienced. For this I offer my most sincere apologies. I understand fully that the value of a forum is not the server or the software, but all the time and effort that are put into the posts contained within by the members.

I have put additional procedures in place to reduce my reliance on tape backups. While I can not say that we will never experience another issue in the future, I can guarantee that this situtation will not occur again. Future backups will be transfered offsite and will not be reliant on tape technology.

The goal right now is the get everything back in place for a Sunday relaunch. Nicky and I will be rebuilding the templates and configurations tonight and tomorrow to make this happen. Unfortunately, everyone will need to create new accounts. If possible, please retain your original account name, as this may help in the event that we are able to migrate the old data back in place. It may take a few more days for all features to be funtional again, specifically member gallerys and banner add rotation.

I appreciate the words of support that Jace has received and relayed to me from all of you. I hope to see everyone back together in the forums very soon.

-Martin

**********************************************************

So, boys and girls... that's where we're at. If you have any questions, please by all means email me at webers@cvalleyDOTnet I'll do my damnedest to get the proper answer back to you.

I apologize for this happening as well. Thank You for all your understanding and support.


Best Regards,
Jace

9 comments:

Fletch said...

Holy Moley! I am IMPRESSED! Way to go, Jace! Way to go Martin! Way to go Nicky! New some bucks, I can cough up a few. Thanks for the note and explanation, Jace, and as you know, when an earthquake decides to strike, there just ain't a lot you or me or anyone else can do about it but hang on!

Anonymous said...

While it is sad and even a little frustrating, each member contains within themselves the things that make TWW the special place that it is. And, we know that it is no fault of our dearly beloved hosts (computers could be added to the things that suck sometimes, not just fleas and tics). Thank you Jace and Martin for working so hard. ShaneG

Ken Belisle said...

Jace....

....thanks for the update and Martin, thanks for the Herculean effort. My personal thought is I would rather see TWW up & running from scratch this weekend than go another couple of weeks without it.

And as Fletch said, if donations would help with the recovery cost, let me know.

Looking forward to see it back on-line soon......

Anonymous said...

All I know is, if we get to create new TWW user accounts, can I call myself Pat Copper? His life sounds way more interesting than mine :-).

Stu in Tokyo said...

Well that sucks, the situation, that is, a real convergence of things gone wrong, looks like Mr. Murphy spent the weekend at your place!

If you guys need some donations to help out, I'm more than willing add to the kitty.

I too would vote for a start from scratch, best to move forward.

Cheers!

Cowguy said...

Thanks for the comments folks. Thanks for the offer Willie and Stu, but rather than get things scattered out, let's just hang on for another day. It's still looking like tomorrow to get lit up, at this point.

Thanks again!

Jace


Also, I've got a few emails left to answer, I'll get to 'em today. Thanks for all the great correspondence!

Anonymous said...

If I leave a comment...do I have to clean it up later? :)

can't wait to get it back up again..bet we lose all our valueable post counts. boo-hoo

Rob

Anonymous said...

I check here every day in fact The Wood Works launches by default on my FireFox browser.

Looking forward to getting things up and running again.

A big thanks to you Jace and Martin!

Bob Genovesi aka BobbyG

gracie said...

We're here.....we're waiting...we're thankful...we love you guys.

Is it June yet?

him and me