I’m seeing an interesting, and seemingly recurring issue with one particular user’s Exchange calendar.  The complain is always that when a new appointment is added to the calendar, it disappears a minute or two later.  The event log on the Exchange Server always shows these errors:

Calendaring agent failed with error code 0×80040215 while saving appointment.

Calendaring agent failed to update the free/busy cache during an appointment save or delete operation.

Calendaring agent failed in message save notification with error 0×800703eb on XXXX@jfbc.org: /Calendar/test.EML.

I’ve tested, researched, tried to reproduce, etc, etc and am at a loss as to what’s causing this.  The only thing I can think of is maybe it’s related to syncing the users Palm Pilot.  Maybe the Palm device is allowing something to be input that Exchange doesn’t like?

I have figured out that there is some corrupt record that exists in the mailbox that’s causing this.  Moving the mailbox to another database and then back clears things up for a while, then it pops back up.  I’m curious if anyone else out there has seen anything like this?

I’ve just started reading through the Psalms, viewing them through the eyes of a leader - applying them to my life.  Here are a few things I’ve learned tonight:

Psalm 1:1-6: God allows me to chose the road I follow.  I can “stand” with sinners or stand with God.  To stand with God, I must find delight in the law of the Lord - by meditating on it day and night.  Awesome - but not always easy.  In fact, it’s sometimes down right hard.

Psalm 3:4,4:3-5,5:1-6: Turn to the Lord first, put all my trust in him, and fully rely on him.  Pray about EVERYTHING.  Pray honestly.  Jeremiah’s Prayer was so honest it’s scary.

Psalm 7:17: Thank the Lord for his rightousness and sing praises to his name.

Psalm 8:1-2,9:1: Give God ALL the glory.  It’s not about me.  Praise him with all of my heart.  Tell of his wonders.

Psalm 10:1-18: God sometimes seemingly moves away - withdraws - often at times when we need him most.  Is he just sitting back watching and not caring?  Or not watching at all?  He doesn’t always stop us when we start to run away.  Has he really stepped back?  No - we just need to move closer to him.

Wow.  Things got really busy and it’s been a couple of weeks since my last blog post.  I was helping with VBS last week, out of town for the weekend, then had a pretty crazy week this week.  It’s been great finally having a nice, slow Saturday to relax.  Hopefully I’ll have more time to sit down and write next week - got lots of cool things going on:

• Continuing to work on better support and management for the growing number of Macs on our network.  Open directory is up and running and integrated with Active Directory!  I’m in the process of upgrading all of our Macs to Leopard and migrating them to the Open Directory system.
• Doing some research and testing to determine whether it’s feasible to use iSCSI as the backend storage for a Final Cut Server implementation - things are looking pretty good so far, even for HD video.
• Still checking out different firewall technologies to see what makes the best sense for our environment.
• Working on building an intranet site in Sharepoint.  Still not convinced Sharepoint is the best solution, but seems to be working OK so far.
• Helping our media team install projectors and sound systems in several classrooms.
• Trying to find time to get some local church IT guys together for a WiFi discussion.
• Trying to find a good time for a vacation.  I think I’m about due to take a little break.

Stay tuned for more updates!

Last week, I put together a list of Network Management Tools that I use on a daily basis and find helpful. Now, I want to spotlight each of those tools. First up on the list: Solarwinds Orion.

Orion is an awesome tool and is the heart of our network management system. It allows me to create visual maps of the network and drill down from the network overview, to a physical location, to device, and then to an individual interface. We can see at a glance any locations, devices, or interfaces with problems.  If any network device goes down or sees packet loss, I immediately get an email.

Orion has really allowed us to be a lot more proactive at managing the network.  In the past, we didn’t know there was a problem until a user called and said they couldn’t access the network.  Then, we had to identify which device had the issue.  With Orion, we can often identify and begin working on a network issue before the end user even knows.

Below are a few screenshots of our environment.

Main page - we can see the entire network at a glance (any location with a problem will have a red exclamation point) as well as a list of current and recent problems:

Individual location - this is out East Cobb location.  A similar map exists for each physical location.  Here, we have every network device located at this location.  The list below the map gives a quick status of each devices and the links between them.  All devices are currently up, which is good, but any device that’s down will have a red exclamation point on it.

Now, let’s look at a device.  Here, we’re looking at our core switch/router.  We get some nice general stats, such as CPU and memory utilization and response time, as well as a lot more detail in the left column: IP address, Device type, host name, IOS version, and last boot time.

Further down the page, we get a list of all of the monitored interfaces on this device, along with their status (up or down), description, and current utilization.  As you can see, the network isn’t very busy today.

From here, we can drill down into an actual interface and see a lot of information about current conditions on that interface, as well as generate custom charts showing historical data.

As you can probably tell, I am really happy with Orion.  There are definitely cheaper products out there, but I haven’t found another product that’s as easy to navigate while giving me this much information about the network.  Once you get more than a couple of switches and servers, a network monitoring tool like this is essential to keep the network up and running at peak performance.

White I was out yesterday, I stopped by the outlet mall for a little while. I didn’t buy anything due to my commitment to the House Pay-Off Spectacular, but I went in a lot of stores and looked around.

Everyone has a “Things That Irk Me” list - things that just irritate you to no end. Well, this is definitely on mine:

In nearly every store I walked into, someone was near the door and greeted me with “Welcome to <store name>, how are you doing today.” I ALWAYS respond, usually with something like “I’m doing great, how are you.” They ALMOST NEVER answer back. I stopped by Sam’s Club today to have my tires balanced. There was a 2 hour wait, so I walked around to several other nearby stores. Same thing today.

Is it asking too much to expect a response back? Have all of these businesses just programmed “Zombies” to repetitively say the same thing and never provide any intelligent response? Is there that little interest in what they are doing? What if the church was the same way? (Unfortunately, some churches are, but I won’t go there).

Anyone else experience the same thing? Does it bother anyone else?

How do we hear the voice of one who is invisible and very rarely uses sound waves to communicate with us? Throughout the Bible, it is made apparent that is is God’s desire to dwell among the people. Adam and Eve heard the voice of God in Genesis 3:8. In John 10:16, Jesus says his sheep hear his voice. Isaiah 30:21 says wherever we go, there will be a voice telling us which way to turn. I could go on and on.

But how do I hear gods voice? In our increasingly “noisy” world, distractions abound. As I sit here and type this, my email and IM clients “Ding” every few minutes, the fan in my laptop constantly slows down and speeds up, another computer is humming over in the corner, the ice maker dumps a batch of ice, cars are passing down the road outside, and every few minutes, I hear the AC unit cycle on and off. And, if that’s bad, then just wait until I set foot in the office on Monday. In our world today, it’s just way too easy get distracted, drift off into another world, and totally tune out God.

So what’s the solution? It depends. There’s no universal answer. For me, I’ve found that getting our for a few hours, or sometimes a weekend, and enjoying nature does wonders. I love to go back in the woods away from all the noise of the cars passing by, where cell phones don’t work - distant from all of the technical toys so prevalent in our “Wired” world - sit down on a rock next to the river, and spend some quality time with God. I’ve had a lot on my mind lately. Lots of unanswered questions and confusion as I walk with the Lord.

It’s absolutely amazing what getting away for just a few hours can do. I spent most of the day yesterday hiking and relaxing away from all of the distractions that are all around me much of the time. Here’s a couple of pictures of my little “outing” with God:

Our recreation department requested that we set them up a few kiosks so that they can handle registering people for events. The idea is to eliminate the paper and extra time involved with on site registrations and have 100% online registration. The requirements are:

• Very portable - easy to roll around to various parts of the building. Our current Children’s checkin kiosks would probably be a be cumbersome for this.
• Wireless - there might not be a network drop everywhere we need to do registration.
• Ability to for user to use it standing up.
• Secure - might not always been 100% supervised, so we need to keep people from doing bad things to the computer, surfing websites other than ours, etc.
• Ability to print confirmation pages to a network printer.

Below is I ended up putting together. Just got the “prototype” assembled last night. I still need to clean up the cabling a bit, but I think it’s going to work.

I’m running Provisio’s SiteKiosk software, which is really cool. It basically locks the user into a full screen browser with limited buttons, no address bar, and access only to specific URL’s. Once the machine is idle for 5 minutes, it will automatically revert back to the home page. A speciall keyboard driver filters out all of the windows special key and combinations. For example, you can’t use the windows key, ctrl-alt-del, alt-tab, and various other combinations. All in all, it’s a really cool setup.

I often get asked “How do you guys handle this?” or “How do you manage that?” Unfortunately, I haven’t found a single tool with a reasonable price that handles every aspect of network management. There are just so many different requirements: Outage reporting, performance monitoring, traffic analysis, device tracking, configuration file management, and IP address management are a few of them.

I’ve implemented a suite of tools that I feel does a really good job. I’m going to start with a list, and follow up every day or two with a spotlight on each product.

• Solarwinds Orion - Orion is the heart of our network management tools. It monitors key interfaces on all of our network devices and provides detailed performance metrics as well as outage notification emails. Everything is presented on nice graphical maps depicting the physical layout of our network.
• Scrutinizer - Scrutinizer is a netflow traffic analyzer. It takes a stream of netlow data from various router interfaces and breaks down traffic by source and destination as well as protocol. I monitor all of our uplinks from each IDF to the core router as well as the point to point circuits to remote sites with scrutinizer.
• Nedi - Nedi is a really cool tool. It goes out and polls every network device every few minutes and retrieves the ARP and CAM tables as well as various other data. It allows me to do some really neat and useful things. For example, if I identify and IP address or MAC address that’s doing something bad, I can search in Nedi and identify exactly which building, switch, and port number that machine is located on.
• Rancid - Rancid manages backing up the configuration of all of our network devices as well as tracking changes. Every switch and router gets polled every hour and, if the configuration has changes, the new configuration is committed to a CVS repository. A web-based viewer allows me to view all of the versions as well as do diffs to determine what changed.
• Infrastructure Search - This is a search utility developed by a friend. It integrates with Rancid and allows me to search every configuration file on every network device for a specific string. This tool is awesome if you’re trying to find, for example, every switch a specific VLAN exists on.
• IPPLan - IPPLan is a IP address management system. It allows me to view each subnet and see what device each IP is assigned for. At last count between all of our internal and external blocks and remote sites, etc., we have 31 different subnets. A tool like IPPLan is essential for keeping up with all of that.
• Future: Snort - I haven’t actually implemented Snort yet, but it’s on my list. Snort is an open source Intrusion Detection and Prevention system.

Over the next few days, I’ll be making a separate post with more details and screenshots of how I’m using each of the above tools.

It’s time for another monthly update on my House Pay-Off Spectacular. I received a tax refund and stimulus payment in May, so I was able to color in quite a few squares this month. It’s exciting to see the progress, even though it involved spending lots of time in Photoshop coloring in squares.

Total Squares: 562
Paid-For Squares: 292 308
Savings to be committed to mortgage: 232
Squares Remaining: 38 22 (wow, didn’t realize how close I actually was until I counted them)

Over the Memorial Day holiday, I had a “VMware Upgrade Party.” I’m not sure it was really a “Party” since I was the lone attendee, but I’ll call it one anyway. I got all of our ESX servers upgraded to the latest build of 3.5, as well as Virtualcenter to the latest 2.5 build. I also added our fourth ESX server, which is the diskless, boot from SAN box I talked about a couple of weeks ago.

I was a little bit hesitant to put the diskless box in production since a bug in the QLogic HBA firmware required me to run their beta or “Limited Release” firmware in order to do Jumbo Frames. So far though, it has been rock solid.

Below, you can see our current VMware environment. I get more and more excited about this every day. I can now have a new machine online in less than 20 minutes without adding any physical hardware. Awesome! We currently have one stand-alone ESX server, jfbc-ecc-esx03 that runs our virtual desktops. The other three servers are in an HA cluster sporting a total of 32GHz of CPU resources and 52GB of RAM. I like it! I hope to be able to add Vmotion and Distributed Resource Scheduling later this year so we can more effectively manage our host resources.