Last December, we implemented 7TB of Equallogic storage as the backbone of our VMware Virtual Infrastructure implementation, as well as to serve as primary storage for our file, Exchange, and SQL servers.  Little did we know that 6 months later we’d be near capacity and shopping for more storage.

Thanks to James at EIS, we now have another 16TB of raw storage online.  Combined with our existing array and considering RAID overhead, we now have just under 15TB of usable iSCSI storage.  I’m excited to have this done!

I absolutely love our Equallogic SAN!  In less than 30 minutes, the new storage array was configured and added into the cluster.  The volumes were automatically distributed and network traffic load balanced among the arrays.  The only complaint I have is that their rack rail system could use some improvement.  Getting the array installed in the rack is the most time consuming part of the entire implementation.

Check it out:

Array was sitting in my office when I arrived on Friday:

Unpacked and ready to be installed.  It has 16 drives with a capacity of 1TB each:

Racked next to it’s little brother.  This is a total of 30 disks and 23TB of raw storage capacity:

Total storage capacity of 14.82TB:

The performance and raw throughput of the Equallogic gear amazes me.  Orion was reporting 534mbps of traffic between the arrays during setup:

Hopefully we will have plenty of space for a while now.  Although, as we move toward implementing Final Cut server and centralized storage for digital video, that may change.

While everyone was away for the holiday Monday, I took the opportunity to upgrade our SAN and ESX servers.  Everything went surprisingly well.

What was really impressive is how fast the Equallogic SAN reboots.  The firmware upgrade was the first reboot since it was installed.  They claimed you could reboot it “live” without causing any problems with the servers, but I had never tested that theory until now.  I was sending it a series of pings every 1 second during the entire process.  I dropped a total of 12 pings during the reboot and the servers never new the storage had just rebooted.  Pretty impressive!  Check this out (I did it from home, hence the 12-15ms latency):

I also migrated all of our ESX servers from version 3.0.2 to 3.5.  For some reason, the HA agent had to be reconfigured on a couple of them, and the ESX firewall decided to block outbound iSCSI traffic on every box after the upgrade.  Other than that, the ESX upgrades went great!

Out first diskless ESX server is no online also.  The QLogic HBA initially wouldn’t connect to our SAN using jumbo frames.  QLogic’s response was to send me their “Beta” or “Limited Release” firmware, which scares me a little.  I have several production VM’s running on that host with no issues though.  I hope to do some benchmarks on VMware Server vs ESX with software iSCSI vs ESX with hardware iSCSI.  Stay tuned for details on that!

I love it when a project goes as planned!

I’ve received several comments and question on my post from a few days ago, “iSCSI Slow? I Think Not.”  The network hardware is critical for peak iSCSI performance.  I think a brief follow up with some details on our network configuration are in order.

We are using a Cisco Catalyst 6506 switch at the core of our network, which handles all of our iSCSI traffic.  The current configuration looks like this:

• (1)  WS-X6K-SUP2-MSFC2 with PFC2 supervisor module
• (2)  WS-X6148A-GE-TX gigabit modules (connects all server and iSCSI devices)
• (1)  WX-X6414-GBIC fiber module (backbone to all of our IDF’s)

All SAN ports are configured for Jumbo Frames and Flow Control.

The servers are HP DL360 G5’s with NC360T Nics.  I just deployed a new ESX server with with a QLogic iSCSI HBA, but I don’t really have any benchmarks on that yet.  I’ll post some details on that once I run some benchmarks.  I’m interested in whether there will be a big performance increase over the ESX software iSCSI initiator.

People love to talk bad about iSCSI, especially “Those Other SAN Vendors” (ie: The Fibre Channel People).  I’ve had a couple of vendors tell me iSCSI is not an enterprise solution and I’d never see over 250Mbps of throughput.  I love proving them wrong.

Check out the images below.  Note that the two transfers below were happening SIMULTANEOUSLY to a single Equallogic PS300 SAN.  That’s a combined throughput of 1.06Gbps! iSCSI rocks!  The key is the network really.  High-end switches with big port buffers, jumbo frames, and flow control are a must.

I have a LOT of really cool and unique projects either in the works or in the planning stages. I can’t believe I get to have this much fun at work! I had a nice chat with John in our media area yesterday about how we can improve our storage, archiving, and workflow in video world.

We produce a LOT of videos. Most of the raw footage these days gets shot directly to hard disk, and archiving and managing all of that digital footage is becoming a big problem. It’s on local disks in edit stations, on removable hard drives, on volumes on our Equallogic SAN - it’s everywhere - and it’s all full or quickly filling up. Then there’s the whole management and workflow issues. How do we find a specific clip or project? How do we allow multiple people to work on the same project simultaneously?

We’ve pretty well decided Final Cut Server is the solution to the content and workflow management portion of the project. It will allow us to group and organize clips with thumbnails and previews, drag and drop directly into final cut, share and collaborate on projects, and even allow Windows machines to view the catalog and watch clips.

Now for the fun part - storing all of that data. How much data are we ultimately talking about? 1TB? 10TB? 100TB? I really don’t know the exact answer to that, but I can tell you this: It’s certainly way more than 1TB and probably way more than 10TB.

The obvious answer is Apple’s XSAN. I’ve definitely explored this, and have implemented and used XSAN in the past. It’s a nice product, but I’m not sure it’s the best solution for our needs. With the Fibre Channel switches, associated cabling, and metadata controllers, the initial implementation cost is high, and, let’s face it: Fibre Channel, although it probably has a few years left, is a dieing technology.

Here’s what I believe I’ve settled on:

Studio Network Solutions has a product called SANmp that allows multiple machines, across platforms, to access iSCSI volumes at the block level. With direct block level iSCSI to each edit station, with appropriate network infrastructure in place - Catalyst 6500 series at the core and probably an HP 2810 series at the edge, I should be able achieve transfer speeds approaching that of Fibre Channel for a fraction of the cost.

Promise has a line of iSCSI SATA arrays that seems like the ultimate solution for our scenario. Their 16 bay unit, loaded with 1TB disks, will give us 16TB of raw storage for a very reasonable price.

The networking side will require pulling a few additional gigE drops and replacing one switch, but most of the network infrastructure is already in place.

For the media asset management side of things, Final Cut Server will run on top of the above infrastructure on an Apple Xserve.

I’m curious if anyone else out there has implemented a similar solution. If so, I’d love to hear from you.

Jason Powell beat me to it.  Check out his post here. And the official Dell/Equallogic press release here.  I’m looking forward to seeing what this can do.  We already use application aware SQL snapshots on our SAN, and it’s great to be able to recover a single database from a volume snapshot.

I’m curious what level of granularity this new tool will allow.  I question how usefully it would be to restore an entire datastore (which I think is what this tool does), since I already have methods to do that.  I guess even restoring at the datastore level, it would be a plus to have it integrated with the SAN.