Last week, I attended the Sonicwall Roadshow in Atlanta for a look at their E-Class firewalls and other products. I’m looking to implement a new firewall solution in the very near future and have pretty much narrowed it down to the Sonicwall E-Class boxes or Cisco. Here’s a few random thoughts about the Sonicwall products:

• The integration with Active Directory and ability to apply firewall rules and policies to users is really cool. As far as I’m aware, the Cisco ASA doesn’t do this. The example they showed was rate limiting YouTube to 30kbps for a group of users. It has a coolness factor to it, but, is it practical? Am I really going to rate limit YouTube for specific users? Most likely not.
• I really like the application level inspection and filtering. Their example was searching for an embedded watermark in a confidential document and preventing it from leaving the network. Examples included looking for it in SMTP and FTP traffic as well as HTTP uploads. I could definitely see a use for this in some businesses. In our environment, it’s not really useful.
• I’m still not a huge fan of their big confusing web interface - althought I will say it has been improved. I’m sure some users prefer the GUI, but I’d much rather have an easy to use command line
• We recently phased out our Sonicwall wireless solution in favor of Xirrus WiFi arrays. The biggest problem we had with the Sonicwall access points is once you got several clients connected, clients would randomly get disconnected and RF strength would fluctuate. I observed the exact same thing happening on the presenter’s laptop at the roadshow. I saw the “Now Connected” dialog pop up 3 times during a 30 minute or so presentation - exact same problem we had. This is not really related to the firewall itself, but thought it was worth mentioning.
• Their VPN client is nice, but there are better VPN clients on the market. We current use Cisco VPN and I am very happy with it.
• It’s expensive - way more than than the Cisco, which is the opposite of what I expected.

The E-Class/NSA series boxes are a huge improvement over the previous generation firewalls. But, comparing it’s features to our needs, and looking at the cost/performance/features ratio, I’m just not convinced it’s right for our environment. Anyone have any further thoughts?

Last week, I met up with several other Church IT guys from the Atlanta area for a discussion on Information Lifecycle Management and backup with Veristor.  We raised a lot of questions and white boarded a pretty scary diagram of how data gets archived and backed up.

In the end, we determined that we need to identify a couple of key time frames:

• RPO, or Recovery Point Objective:  How much data can we afford to lose?
• RTO, or Recovery Time Objective:  How long can we wait to have our data back online?

This is going to take a lot of work from various departments, but I’ve got some initial thoughts.  First, what are our critical apps?  For us, they would be email - communication between our staff and members is critical.  Next would be our Accounting, Payroll, and Membership systems, which are all handled by the same app (Shelby).

So, how long can we be without them?  And what is reasonable given a limited budget?  As much as I’d like to say we can’t lose any data and we need to be back online 10 minutes after a disaster, that is simply not reasonably due to limited financial resources.  We probably could lose a day or so of data on the email and accounting systems and still survive.  Maybe a week on file shares and everything else.  A recovery time of 2 days for account and 1 week on everything else is probably reasonable.

I’ll be evaluating this further, as well as talking to other departments to develop some concrete objects so that we can get a better DR plan in place.

Last week, I got together with a few people from Simple Machines (Open Source project I help manage).  We met up several times to discuss the future of the project, and did a few fun things as well.  On top of that, I had my ESX 3.5 upgrade, the Information Lifecycle meeting with Veristor and other Church IT guys, the SonicWall roadshow, and several other meetings.  Needless to say, I’m behind on my blogging.  I’ll be posting a lot of updates on these various meetings and projects over the next couple of days!

It’s always a bit scary when someone asks you to recover really important data from a week or two ago.  Did the backup run that day successfully?  Did it copy to tape successfully?  (We do disk to disk to tape backup)  Will the restore work?

A little over a week ago, our membership people found about 13,000 “Unassociated” records in the Shelby database.  Under Shelby’s guidance, I did a database backup and we deleted the orphaned records.  Shelby assured us it wouldn’t affect any “good” records.

Well, here we are a week later and apparently a few people are missing, so I’m uploading the current backup and the backup before the pruning so that they can figure out how to restore the deleted people.  Unreliable software is really annoying and Shelby is moving higher and higher up my “Unreliable List.”

While everyone was away for the holiday Monday, I took the opportunity to upgrade our SAN and ESX servers.  Everything went surprisingly well.

What was really impressive is how fast the Equallogic SAN reboots.  The firmware upgrade was the first reboot since it was installed.  They claimed you could reboot it “live” without causing any problems with the servers, but I had never tested that theory until now.  I was sending it a series of pings every 1 second during the entire process.  I dropped a total of 12 pings during the reboot and the servers never new the storage had just rebooted.  Pretty impressive!  Check this out (I did it from home, hence the 12-15ms latency):

I also migrated all of our ESX servers from version 3.0.2 to 3.5.  For some reason, the HA agent had to be reconfigured on a couple of them, and the ESX firewall decided to block outbound iSCSI traffic on every box after the upgrade.  Other than that, the ESX upgrades went great!

Out first diskless ESX server is no online also.  The QLogic HBA initially wouldn’t connect to our SAN using jumbo frames.  QLogic’s response was to send me their “Beta” or “Limited Release” firmware, which scares me a little.  I have several production VM’s running on that host with no issues though.  I hope to do some benchmarks on VMware Server vs ESX with software iSCSI vs ESX with hardware iSCSI.  Stay tuned for details on that!

I love it when a project goes as planned!

I’ve received several comments and question on my post from a few days ago, “iSCSI Slow? I Think Not.”  The network hardware is critical for peak iSCSI performance.  I think a brief follow up with some details on our network configuration are in order.

We are using a Cisco Catalyst 6506 switch at the core of our network, which handles all of our iSCSI traffic.  The current configuration looks like this:

• (1)  WS-X6K-SUP2-MSFC2 with PFC2 supervisor module
• (2)  WS-X6148A-GE-TX gigabit modules (connects all server and iSCSI devices)
• (1)  WX-X6414-GBIC fiber module (backbone to all of our IDF’s)

All SAN ports are configured for Jumbo Frames and Flow Control.

The servers are HP DL360 G5’s with NC360T Nics.  I just deployed a new ESX server with with a QLogic iSCSI HBA, but I don’t really have any benchmarks on that yet.  I’ll post some details on that once I run some benchmarks.  I’m interested in whether there will be a big performance increase over the ESX software iSCSI initiator.

Our new Xserve arrived yesterday.  I got all the initial configuration done and got it racked.  Apple definitely makes some “Pretty” servers.

Over the next few weeks, I’ll be getting Open Directory and Update Services configured and rolled out to all of our Mac workstations.  At some point, we’ll also be installing Final Cut Server.  I’ll be post updates as we get all of this configured.  In the meantime, here’s a few pics:

People love to talk bad about iSCSI, especially “Those Other SAN Vendors” (ie: The Fibre Channel People).  I’ve had a couple of vendors tell me iSCSI is not an enterprise solution and I’d never see over 250Mbps of throughput.  I love proving them wrong.

Check out the images below.  Note that the two transfers below were happening SIMULTANEOUSLY to a single Equallogic PS300 SAN.  That’s a combined throughput of 1.06Gbps! iSCSI rocks!  The key is the network really.  High-end switches with big port buffers, jumbo frames, and flow control are a must.

Yesterday, we had the opportunity to meet with Jill, our new communications director, about how we manage out membership data.  How do we communicate with our members?  Where does the data come from?  What are the problem areas?

We were able to identify at least 8 different types of “Databases” in use other than our Church Management Systen (Shelby).  Yikes!  The next steps are to identify why we are using so many disconnected databases and develop a solution that will meet the needs of the church long term.  It’s going to be a lot of work, but should be fun.

I have a LOT of really cool and unique projects either in the works or in the planning stages. I can’t believe I get to have this much fun at work! I had a nice chat with John in our media area yesterday about how we can improve our storage, archiving, and workflow in video world.

We produce a LOT of videos. Most of the raw footage these days gets shot directly to hard disk, and archiving and managing all of that digital footage is becoming a big problem. It’s on local disks in edit stations, on removable hard drives, on volumes on our Equallogic SAN - it’s everywhere - and it’s all full or quickly filling up. Then there’s the whole management and workflow issues. How do we find a specific clip or project? How do we allow multiple people to work on the same project simultaneously?

We’ve pretty well decided Final Cut Server is the solution to the content and workflow management portion of the project. It will allow us to group and organize clips with thumbnails and previews, drag and drop directly into final cut, share and collaborate on projects, and even allow Windows machines to view the catalog and watch clips.

Now for the fun part - storing all of that data. How much data are we ultimately talking about? 1TB? 10TB? 100TB? I really don’t know the exact answer to that, but I can tell you this: It’s certainly way more than 1TB and probably way more than 10TB.

The obvious answer is Apple’s XSAN. I’ve definitely explored this, and have implemented and used XSAN in the past. It’s a nice product, but I’m not sure it’s the best solution for our needs. With the Fibre Channel switches, associated cabling, and metadata controllers, the initial implementation cost is high, and, let’s face it: Fibre Channel, although it probably has a few years left, is a dieing technology.

Here’s what I believe I’ve settled on:

Studio Network Solutions has a product called SANmp that allows multiple machines, across platforms, to access iSCSI volumes at the block level. With direct block level iSCSI to each edit station, with appropriate network infrastructure in place - Catalyst 6500 series at the core and probably an HP 2810 series at the edge, I should be able achieve transfer speeds approaching that of Fibre Channel for a fraction of the cost.

Promise has a line of iSCSI SATA arrays that seems like the ultimate solution for our scenario. Their 16 bay unit, loaded with 1TB disks, will give us 16TB of raw storage for a very reasonable price.

The networking side will require pulling a few additional gigE drops and replacing one switch, but most of the network infrastructure is already in place.

For the media asset management side of things, Final Cut Server will run on top of the above infrastructure on an Apple Xserve.

I’m curious if anyone else out there has implemented a similar solution. If so, I’d love to hear from you.