I had a conversation yesterday with Bakbone about their NetVault product.  As we’ve moved heavily into virtualization (90% of our infrastructure is virtualized at this point), backup and DR has become a growing challenge.  Ideally, we need to be able to back up entire virtual machines directly from the SAN, with the ability to restore and entire VM, or individual files within a VM.  In addition, properly protecting Active Directory, SQL Server, and Exchange are high priorities.  The ability to do message level restore in Exchange is also somewhat important.

Our aging Backup Exec installation seems to become more and more cumbersome and problematic, and seems to have the common problem of one product trying to do way too much and not doing any one thing exceptionally well. I think it’s time to move into a more enterprise-class product - something more closely tuned to our needs.  NetVault initially seems like a potentially good fit.  If anyone has any experiences with NetVault or has any other recommendations, I’d love to hear from you.

Last week, I met up with several other Church IT guys from the Atlanta area for a discussion on Information Lifecycle Management and backup with Veristor.  We raised a lot of questions and white boarded a pretty scary diagram of how data gets archived and backed up.

In the end, we determined that we need to identify a couple of key time frames:

• RPO, or Recovery Point Objective:  How much data can we afford to lose?
• RTO, or Recovery Time Objective:  How long can we wait to have our data back online?

This is going to take a lot of work from various departments, but I’ve got some initial thoughts.  First, what are our critical apps?  For us, they would be email - communication between our staff and members is critical.  Next would be our Accounting, Payroll, and Membership systems, which are all handled by the same app (Shelby).

So, how long can we be without them?  And what is reasonable given a limited budget?  As much as I’d like to say we can’t lose any data and we need to be back online 10 minutes after a disaster, that is simply not reasonably due to limited financial resources.  We probably could lose a day or so of data on the email and accounting systems and still survive.  Maybe a week on file shares and everything else.  A recovery time of 2 days for account and 1 week on everything else is probably reasonable.

I’ll be evaluating this further, as well as talking to other departments to develop some concrete objects so that we can get a better DR plan in place.

Yesterday, we had the opportunity to meet with Jill, our new communications director, about how we manage out membership data.  How do we communicate with our members?  Where does the data come from?  What are the problem areas?

We were able to identify at least 8 different types of “Databases” in use other than our Church Management Systen (Shelby).  Yikes!  The next steps are to identify why we are using so many disconnected databases and develop a solution that will meet the needs of the church long term.  It’s going to be a lot of work, but should be fun.

I have a LOT of really cool and unique projects either in the works or in the planning stages. I can’t believe I get to have this much fun at work! I had a nice chat with John in our media area yesterday about how we can improve our storage, archiving, and workflow in video world.

We produce a LOT of videos. Most of the raw footage these days gets shot directly to hard disk, and archiving and managing all of that digital footage is becoming a big problem. It’s on local disks in edit stations, on removable hard drives, on volumes on our Equallogic SAN - it’s everywhere - and it’s all full or quickly filling up. Then there’s the whole management and workflow issues. How do we find a specific clip or project? How do we allow multiple people to work on the same project simultaneously?

We’ve pretty well decided Final Cut Server is the solution to the content and workflow management portion of the project. It will allow us to group and organize clips with thumbnails and previews, drag and drop directly into final cut, share and collaborate on projects, and even allow Windows machines to view the catalog and watch clips.

Now for the fun part - storing all of that data. How much data are we ultimately talking about? 1TB? 10TB? 100TB? I really don’t know the exact answer to that, but I can tell you this: It’s certainly way more than 1TB and probably way more than 10TB.

The obvious answer is Apple’s XSAN. I’ve definitely explored this, and have implemented and used XSAN in the past. It’s a nice product, but I’m not sure it’s the best solution for our needs. With the Fibre Channel switches, associated cabling, and metadata controllers, the initial implementation cost is high, and, let’s face it: Fibre Channel, although it probably has a few years left, is a dieing technology.

Here’s what I believe I’ve settled on:

Studio Network Solutions has a product called SANmp that allows multiple machines, across platforms, to access iSCSI volumes at the block level. With direct block level iSCSI to each edit station, with appropriate network infrastructure in place - Catalyst 6500 series at the core and probably an HP 2810 series at the edge, I should be able achieve transfer speeds approaching that of Fibre Channel for a fraction of the cost.

Promise has a line of iSCSI SATA arrays that seems like the ultimate solution for our scenario. Their 16 bay unit, loaded with 1TB disks, will give us 16TB of raw storage for a very reasonable price.

The networking side will require pulling a few additional gigE drops and replacing one switch, but most of the network infrastructure is already in place.

For the media asset management side of things, Final Cut Server will run on top of the above infrastructure on an Apple Xserve.

I’m curious if anyone else out there has implemented a similar solution. If so, I’d love to hear from you.

Currently, our network at JFBC is about 15% Mac. One of the big ongoing projects I’ve been working on is better integration and management of the growing number of Macs in our environment. We currently leverage Active Directory for single signon, but, beyond that, there are no real management tools in place for Macs.

Some things are possible by extending the Active Directory schema to add some of the apple-specific LDAP attributes. However, this moves the AD environment into a somewhat “unsupported” configuration and still doesn’t provide for full control when it comes to Mac management.

The best way to fully manage the Mac clients - including centralized update management and general settings, including appearance, shortcuts, scripting, etc. is through the use of Apple’s Open Directory system. There was definitely some effort put forth on Apple’s part here, because Open Directory can fully integrate with Active Directory. Basically, AD gets used for authentication, then AD users and groups can be linked to OD groups. Specific management settings can then be applied to the OD groups.

I’ve just ordered a new Apple Xserve to handle this task, which should arrive next week. I’m excited about being able to take integration and management of our Mac environment to the next level.

Other Mac stuff on my radar:

• OS X Leopard deployment (Jonathan has agreed be my next victim beta tester).
• Office 2008 deployment.
• Possible Final Cut Server implementation (Already briefly discussed with our media team, will be exploring this further, including storage requirements).
• Migration of our closed circuit TV announcements from PowerPoint on Windows to Keynote on Mac (Currently working with our communications team on this).

Expect lots of Mac related posts in the coming weeks/months!

Tony’s post “Should I pursue unexcellent?” got my mind going. The key here is that excellence does not equal perfection. If we expect perfection out of our ministry, we will fail every time and be miserable in the process.

So what’s the difference? I like to think of it this way: Perfection is the act of being perfect while excellence is striving to be perfect. Perfection is being absolutely flawless, without error, never making any mistakes. Excellence is allowing our God-given talents to express themselves to the max.

So, excellence = mediocrity? Absolutely not! The bible commands us to strive to be perfect (excellence). This is apparent over and over - Philippians 1:10, 3:12-13, 4:8, 1 Peter 1:16. We will not achieve perfection until we are eternally united with God in Heaven, but we are to press on towards this goal. That is excellence.

Here are a few other random thoughts:

• Perfection is a fear of mistakes while excellence sees opportunities for improvement.
• Perfection is attempting to be in control while excellence is allowing god to be in control.
• Perfection is setting unreasonably standards will beyond reach. Excellence is setting high and perhaps difficult to attain, but attainable standards.
• Perfection can lead to misery and failure - it’s simply not attainable here on Earth. Excellence, however, leads to fulfillment and greatness.

Now, we do have to be careful. What we don’t want is to use excellence as an excuse for mediocrity. It’s easy to say “It can never be perfect, so this will have to do.” Sorry, but that “Ain’t gonna cut it.” We should never settle for “Good” when we can have “Great.”

The bottom line is: We must recognize that excellence is not perfection, but it’s also not mediocrity. We are to give our best, but also set attainable goals and recognize out limits.

I’ve been evaluating our current firewall situation and came to the following conclusion: We have too many products from too many vendors costing too much money and causing too many headaches.

Currently, we use a Watchguard Firebox X700 for our internal network and DMZ. It’s fine when it works, but gets flaky sometimes. It will randomly stop passing FTP traffic, or you add a new rule and it won’t actually work until you reboot. Overall, I’m not impressed.

Inside the Watchguard sites a Barracuda Web Filter. I like this box - it works great and causes very few issues. They do silly stuff sometimes like adding Northpoint.org to the “Spam” category, but, overall, I’m happy.

For our public network, there is a Sonicwall Pro 3060 running Sonicwall’s own content filtering service. The Sonicwall is a nice box, but has it’s issues with occasionally locking up and other weird stuff. I’ve used them in the past and came to the conclusion that it’s a great small office firewall, but really not an enterprise-class solution. The yearly support cost plus the yearly content filter license gets expensive too.

For VPN, we use a Cisco ASA 5510. This replaced the Watchguard VPN (with is absolutely horrible) and provides VPN access for laptops and a few remote sites (5505’s at the remote sites). I love the ASA firewalls. They are rock solid on both stability and performance and, in my opinion, easier to configure. This is definitely a keeper. Ironically, of the three boxes, the Cisco has the lowest yearly support cost.

I’m sure there were good reasons for implementing each product at the time, but everything but the Cisco was here before my time, so I don’t know all the details.

In putting it all together, this is what I’m seeing:

• We’re paying for subscriptions to two different content filtering services - one for the internal network and one for the public network.
• We have support contracts with 3 different vendors.
• There are some stability problems.
• Learning curve involved with multiple products.

Most of those contracts are up for some renewal, so now is the time for change. My thought is to get another ASA 5510 use it’s multiple interfaces to attach the internet network, the DMZ, and the wireless to one firewall. Then, hang the Barracuda Web Filter off another interface and redirect all outbound http traffic to it via WCCP.

It seems almost two simple. Once firewall for all the network segments, one content filter for everything, one less piece of hardware, more stability. It just makes sense all around. If anyone has an thoughts on this, I’d love to hear them.

Tony made an interesting comment on my Remote Access Post from a few days ago. He has a good point, and I think it’s worth visiting. We do we give remote access to and from what computers. Is it a good idea to allow them access from their personal computers? I have had the same thought, and that is the primary reason I’m not already doing it. Here’s a few thoughts:

Generally, only a user with a church supplied laptop would be given VPN rights. If the user has been granted the right to log in via VPN, though, can I really control what machine they do it from? I really can’t. All they need is the Cisco VPN client and a few configuration details, so, in theory, a tech-savy user could access the VPN from any computer.

• Is it any different from a rogue computer being physically plugged into the network? No, it’s really not. Now, random machines being attached to the network is definitely not something I promote or desire. But, it would take very extreme measures and a lot of expense to stop it. 802.1x is a possibility, but, beyond that, it would require some sort of centralized MAC Address based authentication. This exists from a few vendors, but isn’t cheap. Bottom line is, it’s not easy or cheap to keep rogue machines out completely.
• Putting costs and implementation issues aside, what impact would it have on ministry to implement the above? Are there legitimate reasons for someone to attach a “rogue” machine to the network? In general, no, but there are some exceptions.
• We live in an increasingly “connected” and mobile society. Ministry is no exception. Increasingly, being on the cutting edge of technology is a requirement of our ministry. It is absolutely critical that we enable our staff to perform their duties without being physically present in the office.

So, with the above in mind, I’ve placed a greater focus on keeping our internal defenses in line. Here’s a few actions I’ve taken or plan to take:

• Windows firewall is enabled on all workstations via group policy and no programs are allowed to create exceptions. There are only a handful of ports allowed.
• This is a no-brainer, but centrally managed anti-virus is in place on all internal machines.
• SMTP is not allowed outbound from anywhere on the network, with the exception of our exchange servers. This limits the scope of damage should a machine with a mass-mailing worm show up on the network.
• Access to network file shares is very carefully controlled. User accounts do not have access to anything they do not specifically needs to access for their job function.

After putting a lot of thought into it, I’ve come to the conclusion that the benefits to allowing VPN access outweigh the potential negative impact of not allowing it. I would rather allow limited access via other methods of possible, which is why I’m exploring Terminal Services Web Access combined with RemoteApp. But, if for some reason the Terminal Services solution does not work out, I believe VPN is an acceptable fall-back.

I attended the Microsoft product launch event in Atlanta yesterday.  Been traveling all day today, so just not getting a chance to type up a post about it. Overall, it was a great experience.  Got to see and learn about lots of cool new features in Window Server 2008 that I haven’t had a chance to test yet.  Also talked to several interesting vendors.  Here’s a few random thoughts:

• Freebies are always nice.  I walked away with a full copy of Windows Vista Ultimate and limited or trial versions of several other products.
• Overall, the speakers were really good.
• The exhibit hall was like way crowded in the morning.  Hard to have a serious conversation with the vendors do to the number of people present.  Got a little better later in the day.
• Network access protection is interesting - can check certain aspects of a machine, such as antivirus and firewall status and only allow access to servers if certain criteria are met.  Optionally, traffic to servers containing sensitive data can be IPSec protected.  Awesome concepts, but I have a feeling it’s going to be difficult to manage and create headaches for users and IT staff.  I might play with it a bit if I can find time.
• TS RemoteApp is really nice.  We already have a couple of servers running it and will roll it out in our production environment really soon for Shelby and EMS.
• I really, really like Terminal Server Gateway and Web Access features.  Combined with RemoteApp, it seems like the perfect solution for users to access certain apps from their home computers.  Basically, the user logs into a web site and it presented with a Windows desktop looking screen with icons for apps and servers they can click to connect to.  All the traffic gets tunneled over SSL, which makes me less concerned than opening RDP to the outside.  It can also be hidden behind an ISA server (which we already use for remote Exchange access) for additional security.  I really think this is the answer to some of my remote access woes.
• I thought the web server session was boring.
• MS has come a long way with their virtualization products.  In my opinion, it’s still not up to par with VMware though.  There’s still no live migration ability.  There’s still a host OS involved (I think about how many times I have to reboot windows servers vs how I never have to reboot ESX).  There’s no way to resize a virtual hard disk (seems really odd to me, VMware has had this for years).  Give it a couple more years, and HyperV might be the way to go.  I’m not convinced yet though.

Yesterday was an incredible day. We started out at Perimeter where Tony showed me his intranet site for displaying Shelby data as well as Syncra. Jarrod Barden from Newspring Church, Errol Conner from Fellowship Technologies, and Justin Moore joined us (and probably others I’m forgetting. We did a quick tour of Perimeter, then headed over to Garrison’s for lunch.

After lunch, we headed over to First Baptist Atlanta and met up with several others. Justin did a great presentation on Asterisk. You can see his slides and calculator here. I’ll leave the transcripts to one of the guys who took really good notes.

After the Asterisk discussion, we headed over to Paetech for a discussion with some of their team as well as a switch tour. It was cool to get to see their behind the scenes operations as well as chat with some of their team.

Finally, Errol stopped by Johnson Ferry for a little while and I had the opportunity to have a great conversation with him about managing our data. I stayed at the office a while after Errol left to finish up a few things. It was a long day, but an awesome one. Wouldn’t it be great if we could all get together more often?