I’m excited about this month’s House Pay-Off Spectacular.  The green and pink are actually touching now!  I’m praying and looking forward to being able to go all green very, very soon!  Hopefully in the next month.  At this point, I’m just waiting on some funds in CD’s to become available.

This is awesome!

Total Squares: 562
Paid-For Squares: 323 336
Savings to be committed to mortgage: 232 219
Squares Remaining: 7 0

If you are not aware yet, a major bug has been revealed in ESX 3.5 and ESXi 3.5 Update 2.  Apparently, the beta was coded to expire on August 12, 2008 and this code failed to be removed from the actual release.  Details are available in the VMware Knowledge Base and This Topic on their forums.  You might also checkout This Post on Matthew Marlow’s blog for more information.

On the morning of the 12th, I was greeted with several errors like this one in the logs for our ESX cluster:

VMware finally released the patch really late Tuesday night, which, unfortunately kept me up most of the night getting our cluster patched.  It involved setting back the clock on all of the hosts so Vmotion would work, manually migrating VM’s off of a host, going into maintenance mode, applying the patch via the command line, then migrating the VM’s back.

VMware is one of my favorite companies and it know for delivering rock-sold, enterprise-class products, so it really disappoints me that they would let something like this slip through the cracks.  Imagine how many hundreds of thousands (Maybe millions?) of VM’s this affected.  They do seem to be committed to fixing their mistake and making things right.  You can check out the Letter From Their CEO for more info.

I had a conversation yesterday with Bakbone about their NetVault product.  As we’ve moved heavily into virtualization (90% of our infrastructure is virtualized at this point), backup and DR has become a growing challenge.  Ideally, we need to be able to back up entire virtual machines directly from the SAN, with the ability to restore and entire VM, or individual files within a VM.  In addition, properly protecting Active Directory, SQL Server, and Exchange are high priorities.  The ability to do message level restore in Exchange is also somewhat important.

Our aging Backup Exec installation seems to become more and more cumbersome and problematic, and seems to have the common problem of one product trying to do way too much and not doing any one thing exceptionally well. I think it’s time to move into a more enterprise-class product - something more closely tuned to our needs.  NetVault initially seems like a potentially good fit.  If anyone has any experiences with NetVault or has any other recommendations, I’d love to hear from you.

Last December, we implemented 7TB of Equallogic storage as the backbone of our VMware Virtual Infrastructure implementation, as well as to serve as primary storage for our file, Exchange, and SQL servers.  Little did we know that 6 months later we’d be near capacity and shopping for more storage.

Thanks to James at EIS, we now have another 16TB of raw storage online.  Combined with our existing array and considering RAID overhead, we now have just under 15TB of usable iSCSI storage.  I’m excited to have this done!

I absolutely love our Equallogic SAN!  In less than 30 minutes, the new storage array was configured and added into the cluster.  The volumes were automatically distributed and network traffic load balanced among the arrays.  The only complaint I have is that their rack rail system could use some improvement.  Getting the array installed in the rack is the most time consuming part of the entire implementation.

Check it out:

Array was sitting in my office when I arrived on Friday:

Unpacked and ready to be installed.  It has 16 drives with a capacity of 1TB each:

Racked next to it’s little brother.  This is a total of 30 disks and 23TB of raw storage capacity:

Total storage capacity of 14.82TB:

The performance and raw throughput of the Equallogic gear amazes me.  Orion was reporting 534mbps of traffic between the arrays during setup:

Hopefully we will have plenty of space for a while now.  Although, as we move toward implementing Final Cut server and centralized storage for digital video, that may change.

Matt McCook from Xirrus has offered to do a presentation on high-density WiFi and how their products can meet these needs starting at noon on July 17 at JFBC.  We have been using Xirrus products at JFBC for just over a year now and have been extremely happy with the performance and reliability of the Xirrus solution.  You can read about our initial deployment on the Xirrus Site.  I’m excited to be able to have Matt come and share their solution with other Atlanta area Church IT guys.  If you have a need for a high-density WiFi solution and would like to join us, please drop me a note at derek.schwab@jfbc.org or post a comment.

Wow!  It’s time for the July update on my House Pay-Off Spectacular and am PUMPED about it.  It still feels kind of silly to get so excited about coloring in squares in Photoshop, but I’m thrilled with what God has givin me.  He has truly blessed me with way more than I deserve.

It looks like there will actually end up being a couple of unexpected months of overlap between the green and the pink due to the fact some of the pink $$$ is in CD’s and not immediately accessible.  It’s still awesome though!

Total Squares: 562
Paid-For Squares: 308 323
Savings to be committed to mortgage: 232
Squares Remaining: 22 7

I’m seeing an interesting, and seemingly recurring issue with one particular user’s Exchange calendar.  The complain is always that when a new appointment is added to the calendar, it disappears a minute or two later.  The event log on the Exchange Server always shows these errors:

Calendaring agent failed with error code 0×80040215 while saving appointment.

Calendaring agent failed to update the free/busy cache during an appointment save or delete operation.

Calendaring agent failed in message save notification with error 0×800703eb on XXXX@jfbc.org: /Calendar/test.EML.

I’ve tested, researched, tried to reproduce, etc, etc and am at a loss as to what’s causing this.  The only thing I can think of is maybe it’s related to syncing the users Palm Pilot.  Maybe the Palm device is allowing something to be input that Exchange doesn’t like?

I have figured out that there is some corrupt record that exists in the mailbox that’s causing this.  Moving the mailbox to another database and then back clears things up for a while, then it pops back up.  I’m curious if anyone else out there has seen anything like this?

I’ve just started reading through the Psalms, viewing them through the eyes of a leader - applying them to my life.  Here are a few things I’ve learned tonight:

Psalm 1:1-6: God allows me to chose the road I follow.  I can “stand” with sinners or stand with God.  To stand with God, I must find delight in the law of the Lord - by meditating on it day and night.  Awesome - but not always easy.  In fact, it’s sometimes down right hard.

Psalm 3:4,4:3-5,5:1-6: Turn to the Lord first, put all my trust in him, and fully rely on him.  Pray about EVERYTHING.  Pray honestly.  Jeremiah’s Prayer was so honest it’s scary.

Psalm 7:17: Thank the Lord for his rightousness and sing praises to his name.

Psalm 8:1-2,9:1: Give God ALL the glory.  It’s not about me.  Praise him with all of my heart.  Tell of his wonders.

Psalm 10:1-18: God sometimes seemingly moves away - withdraws - often at times when we need him most.  Is he just sitting back watching and not caring?  Or not watching at all?  He doesn’t always stop us when we start to run away.  Has he really stepped back?  No - we just need to move closer to him.

Wow.  Things got really busy and it’s been a couple of weeks since my last blog post.  I was helping with VBS last week, out of town for the weekend, then had a pretty crazy week this week.  It’s been great finally having a nice, slow Saturday to relax.  Hopefully I’ll have more time to sit down and write next week - got lots of cool things going on:

• Continuing to work on better support and management for the growing number of Macs on our network.  Open directory is up and running and integrated with Active Directory!  I’m in the process of upgrading all of our Macs to Leopard and migrating them to the Open Directory system.
• Doing some research and testing to determine whether it’s feasible to use iSCSI as the backend storage for a Final Cut Server implementation - things are looking pretty good so far, even for HD video.
• Still checking out different firewall technologies to see what makes the best sense for our environment.
• Working on building an intranet site in Sharepoint.  Still not convinced Sharepoint is the best solution, but seems to be working OK so far.
• Helping our media team install projectors and sound systems in several classrooms.
• Trying to find time to get some local church IT guys together for a WiFi discussion.
• Trying to find a good time for a vacation.  I think I’m about due to take a little break.

Stay tuned for more updates!

Last week, I put together a list of Network Management Tools that I use on a daily basis and find helpful. Now, I want to spotlight each of those tools. First up on the list: Solarwinds Orion.

Orion is an awesome tool and is the heart of our network management system. It allows me to create visual maps of the network and drill down from the network overview, to a physical location, to device, and then to an individual interface. We can see at a glance any locations, devices, or interfaces with problems.  If any network device goes down or sees packet loss, I immediately get an email.

Orion has really allowed us to be a lot more proactive at managing the network.  In the past, we didn’t know there was a problem until a user called and said they couldn’t access the network.  Then, we had to identify which device had the issue.  With Orion, we can often identify and begin working on a network issue before the end user even knows.

Below are a few screenshots of our environment.

Main page - we can see the entire network at a glance (any location with a problem will have a red exclamation point) as well as a list of current and recent problems:

Individual location - this is out East Cobb location.  A similar map exists for each physical location.  Here, we have every network device located at this location.  The list below the map gives a quick status of each devices and the links between them.  All devices are currently up, which is good, but any device that’s down will have a red exclamation point on it.

Now, let’s look at a device.  Here, we’re looking at our core switch/router.  We get some nice general stats, such as CPU and memory utilization and response time, as well as a lot more detail in the left column: IP address, Device type, host name, IOS version, and last boot time.

Further down the page, we get a list of all of the monitored interfaces on this device, along with their status (up or down), description, and current utilization.  As you can see, the network isn’t very busy today.

From here, we can drill down into an actual interface and see a lot of information about current conditions on that interface, as well as generate custom charts showing historical data.

As you can probably tell, I am really happy with Orion.  There are definitely cheaper products out there, but I haven’t found another product that’s as easy to navigate while giving me this much information about the network.  Once you get more than a couple of switches and servers, a network monitoring tool like this is essential to keep the network up and running at peak performance.