Message from NSC CEO

To Members and Guests of NavySEALs.com:

Many of you have wondered what the heck happened to the NavySEALs.com community site the last few days. Now that we are back online I can give you some insight.

We experienced a combination of human and technical errors that combined to create our worst technical outage since we went live in 1997. It went something like this: our server hard drives hit their capacity in early June, so we scheduled an upgrade to larger capacity. Our hosting company ghosted the original hard drives, then installed the new drives last Saturday. The data transfer of the MySQL database to the new drives failed, so the hosting company re-installed the older drives to bring the site back up. Well, unfortunately the old drives, which had been running at capacity for a couple weeks, failed to work when they were reinstalled.

The only option left was a complete system restore and a data migration from a back-up of the system. Murphy struck again, and the hosting company inadvertently restored a back-up snapshot that was taken during this whole incredible screw-up - it was a bad back-up and poof - 17 more hours of work down the drain. Finally we were able to get a good backup (from June 27) restored to the system and were up and running early Wednesday morning.

I suppose that I re-learned two valuable lessons from this experience...both that I learned in the SEAL Teams but failed to apply in this situation (I'm pleading technological ignorance!):

1. That "one is none, and two is one." we need to create a redundant system to fall back on when the primary fails, and
2. That "no plan survives contact with the enemy", so have an E & E plan. We will develop a disaster recovery plan that covers all envisioned contingencies.

It is valuable to think like a SEAL in business and in life. I have learned that valuable lesson again here at NavySEALs.com. I apologize for any inconvenience this event may have caused and thank you for your support!

Sincerely,
Mark Divine
CEO U.S. Tactical and NavySEALs.com

del.icio.us