The last year or so, we’ve been moving full force at doing lots of cleanup and building an enterprise-class infrastructure.  There’s still a ways to go, but now it’s time to develop a solid IT strategy.  I’ll be making several updates along that journey, but one of the bigs things that’s came up is what do we do when things don’t go right?  A big issue is: What do we do if there’s a major power outage and we have to shut everything down and bring it back up?

Now, last year, we had the opportunity to install a 15KVA UPS that’s capable of running our entire server farm, network core, and phone system for about 2 hours.  We rarely have an outage that long, so we haven’t had to do a full shutdown/startup in quite a while.  During the last year or so, as we’ve implemented technologies such as SAN storage and Virtualization, it seems our infrastructure has gotten considerably more complex with lots of systems being interdependent on each other.

Here’s a short list of dependencies that come to mind:

• Exchange, Blackberry Server, SQL Server, Virtual Center, and VPN all require Active Directory to be up.
• Virtual Center obviously needs the ESX servers up to function.
• Virtual Center and Blackberry Server use the SQL server.
• Blackberry Server depends on Exchange being up.
• The ESX cluster, Exchange, and SQL server all require SAN storage.
• The SAN requires the core switches be functional before it comes online.

As you can see, the dependencies quickly get complex.  If you have to shutdown everything, what order do you do it in?  How do you bring it back up?  I’ll be documenting all of this over the coming weeks and hopefully actually doing a test at some point.  More updates to come.