Matt McCook from Xirrus has offered to do a presentation on high-density WiFi and how their products can meet these needs starting at noon on July 17 at JFBC.  We have been using Xirrus products at JFBC for just over a year now and have been extremely happy with the performance and reliability of the Xirrus solution.  You can read about our initial deployment on the Xirrus Site.  I’m excited to be able to have Matt come and share their solution with other Atlanta area Church IT guys.  If you have a need for a high-density WiFi solution and would like to join us, please drop me a note at derek.schwab@jfbc.org or post a comment.

Last week, I attended the Sonicwall Roadshow in Atlanta for a look at their E-Class firewalls and other products. I’m looking to implement a new firewall solution in the very near future and have pretty much narrowed it down to the Sonicwall E-Class boxes or Cisco. Here’s a few random thoughts about the Sonicwall products:

• The integration with Active Directory and ability to apply firewall rules and policies to users is really cool. As far as I’m aware, the Cisco ASA doesn’t do this. The example they showed was rate limiting YouTube to 30kbps for a group of users. It has a coolness factor to it, but, is it practical? Am I really going to rate limit YouTube for specific users? Most likely not.
• I really like the application level inspection and filtering. Their example was searching for an embedded watermark in a confidential document and preventing it from leaving the network. Examples included looking for it in SMTP and FTP traffic as well as HTTP uploads. I could definitely see a use for this in some businesses. In our environment, it’s not really useful.
• I’m still not a huge fan of their big confusing web interface - althought I will say it has been improved. I’m sure some users prefer the GUI, but I’d much rather have an easy to use command line
• We recently phased out our Sonicwall wireless solution in favor of Xirrus WiFi arrays. The biggest problem we had with the Sonicwall access points is once you got several clients connected, clients would randomly get disconnected and RF strength would fluctuate. I observed the exact same thing happening on the presenter’s laptop at the roadshow. I saw the “Now Connected” dialog pop up 3 times during a 30 minute or so presentation - exact same problem we had. This is not really related to the firewall itself, but thought it was worth mentioning.
• Their VPN client is nice, but there are better VPN clients on the market. We current use Cisco VPN and I am very happy with it.
• It’s expensive - way more than than the Cisco, which is the opposite of what I expected.

The E-Class/NSA series boxes are a huge improvement over the previous generation firewalls. But, comparing it’s features to our needs, and looking at the cost/performance/features ratio, I’m just not convinced it’s right for our environment. Anyone have any further thoughts?

I’ve received several comments and question on my post from a few days ago, “iSCSI Slow? I Think Not.”  The network hardware is critical for peak iSCSI performance.  I think a brief follow up with some details on our network configuration are in order.

We are using a Cisco Catalyst 6506 switch at the core of our network, which handles all of our iSCSI traffic.  The current configuration looks like this:

• (1)  WS-X6K-SUP2-MSFC2 with PFC2 supervisor module
• (2)  WS-X6148A-GE-TX gigabit modules (connects all server and iSCSI devices)
• (1)  WX-X6414-GBIC fiber module (backbone to all of our IDF’s)

All SAN ports are configured for Jumbo Frames and Flow Control.

The servers are HP DL360 G5’s with NC360T Nics.  I just deployed a new ESX server with with a QLogic iSCSI HBA, but I don’t really have any benchmarks on that yet.  I’ll post some details on that once I run some benchmarks.  I’m interested in whether there will be a big performance increase over the ESX software iSCSI initiator.

People love to talk bad about iSCSI, especially “Those Other SAN Vendors” (ie: The Fibre Channel People).  I’ve had a couple of vendors tell me iSCSI is not an enterprise solution and I’d never see over 250Mbps of throughput.  I love proving them wrong.

Check out the images below.  Note that the two transfers below were happening SIMULTANEOUSLY to a single Equallogic PS300 SAN.  That’s a combined throughput of 1.06Gbps! iSCSI rocks!  The key is the network really.  High-end switches with big port buffers, jumbo frames, and flow control are a must.

I’ve been evaluating our current firewall situation and came to the following conclusion: We have too many products from too many vendors costing too much money and causing too many headaches.

Currently, we use a Watchguard Firebox X700 for our internal network and DMZ. It’s fine when it works, but gets flaky sometimes. It will randomly stop passing FTP traffic, or you add a new rule and it won’t actually work until you reboot. Overall, I’m not impressed.

Inside the Watchguard sites a Barracuda Web Filter. I like this box - it works great and causes very few issues. They do silly stuff sometimes like adding Northpoint.org to the “Spam” category, but, overall, I’m happy.

For our public network, there is a Sonicwall Pro 3060 running Sonicwall’s own content filtering service. The Sonicwall is a nice box, but has it’s issues with occasionally locking up and other weird stuff. I’ve used them in the past and came to the conclusion that it’s a great small office firewall, but really not an enterprise-class solution. The yearly support cost plus the yearly content filter license gets expensive too.

For VPN, we use a Cisco ASA 5510. This replaced the Watchguard VPN (with is absolutely horrible) and provides VPN access for laptops and a few remote sites (5505’s at the remote sites). I love the ASA firewalls. They are rock solid on both stability and performance and, in my opinion, easier to configure. This is definitely a keeper. Ironically, of the three boxes, the Cisco has the lowest yearly support cost.

I’m sure there were good reasons for implementing each product at the time, but everything but the Cisco was here before my time, so I don’t know all the details.

In putting it all together, this is what I’m seeing:

• We’re paying for subscriptions to two different content filtering services - one for the internal network and one for the public network.
• We have support contracts with 3 different vendors.
• There are some stability problems.
• Learning curve involved with multiple products.

Most of those contracts are up for some renewal, so now is the time for change. My thought is to get another ASA 5510 use it’s multiple interfaces to attach the internet network, the DMZ, and the wireless to one firewall. Then, hang the Barracuda Web Filter off another interface and redirect all outbound http traffic to it via WCCP.

It seems almost two simple. Once firewall for all the network segments, one content filter for everything, one less piece of hardware, more stability. It just makes sense all around. If anyone has an thoughts on this, I’d love to hear them.

Tony made an interesting comment on my Remote Access Post from a few days ago. He has a good point, and I think it’s worth visiting. We do we give remote access to and from what computers. Is it a good idea to allow them access from their personal computers? I have had the same thought, and that is the primary reason I’m not already doing it. Here’s a few thoughts:

Generally, only a user with a church supplied laptop would be given VPN rights. If the user has been granted the right to log in via VPN, though, can I really control what machine they do it from? I really can’t. All they need is the Cisco VPN client and a few configuration details, so, in theory, a tech-savy user could access the VPN from any computer.

• Is it any different from a rogue computer being physically plugged into the network? No, it’s really not. Now, random machines being attached to the network is definitely not something I promote or desire. But, it would take very extreme measures and a lot of expense to stop it. 802.1x is a possibility, but, beyond that, it would require some sort of centralized MAC Address based authentication. This exists from a few vendors, but isn’t cheap. Bottom line is, it’s not easy or cheap to keep rogue machines out completely.
• Putting costs and implementation issues aside, what impact would it have on ministry to implement the above? Are there legitimate reasons for someone to attach a “rogue” machine to the network? In general, no, but there are some exceptions.
• We live in an increasingly “connected” and mobile society. Ministry is no exception. Increasingly, being on the cutting edge of technology is a requirement of our ministry. It is absolutely critical that we enable our staff to perform their duties without being physically present in the office.

So, with the above in mind, I’ve placed a greater focus on keeping our internal defenses in line. Here’s a few actions I’ve taken or plan to take:

• Windows firewall is enabled on all workstations via group policy and no programs are allowed to create exceptions. There are only a handful of ports allowed.
• This is a no-brainer, but centrally managed anti-virus is in place on all internal machines.
• SMTP is not allowed outbound from anywhere on the network, with the exception of our exchange servers. This limits the scope of damage should a machine with a mass-mailing worm show up on the network.
• Access to network file shares is very carefully controlled. User accounts do not have access to anything they do not specifically needs to access for their job function.

After putting a lot of thought into it, I’ve come to the conclusion that the benefits to allowing VPN access outweigh the potential negative impact of not allowing it. I would rather allow limited access via other methods of possible, which is why I’m exploring Terminal Services Web Access combined with RemoteApp. But, if for some reason the Terminal Services solution does not work out, I believe VPN is an acceptable fall-back.

I just installed the Xirrus WiFi monitor gadget and thought it was kind of cool. Shows all available access points and their relative location on a “Radar Screen” with signal strength, SSID, MAC Address, etc. Nothing to see here at the house but my own AP. Will have to play with it some more at the office.

As we become more and more mobile remote access becomes more and more important.  It’s easy for people who have church-issued laptops.  We have a Cisco IPSec VPN that works great.

But, what about users without laptops who need access to certain apps and services?   There’s several options available, but I’m not convinced any of them are great:

• Connect to VPN and install apps (Shelby, EMS, etc) on home computer.  Obviously, this is very difficult to support and can be slow.
• Connect to VPN and Remote Desktop to their own computer.  I have a couple of users who do this now and it works.  Maybe it’s the best way to handle it since once they’re in, the experience is the same as at the office.  It usually requires a phone call to walk the user through the RDP setup, but it’s not too bad to deal with
• Terminal Services gateway.  I have reservations about opening any MS product up to the internet.  I guess it could be hidden behind an ISA server with RADIUS authentication - we already do this for OWA access to exchange.  Combined with WIndows 2008 RemoteAPP, this could be a very good option, especially since it wouldn’t require a VPN client.  May be a security concern.
• VPN client + RemoteApps - would be easy for the user - I just need to give them a couple of RDP files.  What about accessing Word, Excel, etc?
• Cisco WebVPN - this is cool because it allows the user to log into a web interface and access CIFS file shares.  A bit of a pain to setup and manage though, and doesn’t really allow for the user to access apps.
• VPN client + terminal server - eliminates need to RDP to a workstation, but user may need apps not available on the terminal server.

At this point, I’m kind of leaning toward just allowing users to RDP into their own workstation over a VPN connection.  Anyone have any better ideas?

The last year or so, we’ve been moving full force at doing lots of cleanup and building an enterprise-class infrastructure.  There’s still a ways to go, but now it’s time to develop a solid IT strategy.  I’ll be making several updates along that journey, but one of the bigs things that’s came up is what do we do when things don’t go right?  A big issue is: What do we do if there’s a major power outage and we have to shut everything down and bring it back up?

Now, last year, we had the opportunity to install a 15KVA UPS that’s capable of running our entire server farm, network core, and phone system for about 2 hours.  We rarely have an outage that long, so we haven’t had to do a full shutdown/startup in quite a while.  During the last year or so, as we’ve implemented technologies such as SAN storage and Virtualization, it seems our infrastructure has gotten considerably more complex with lots of systems being interdependent on each other.

Here’s a short list of dependencies that come to mind:

• Exchange, Blackberry Server, SQL Server, Virtual Center, and VPN all require Active Directory to be up.
• Virtual Center obviously needs the ESX servers up to function.
• Virtual Center and Blackberry Server use the SQL server.
• Blackberry Server depends on Exchange being up.
• The ESX cluster, Exchange, and SQL server all require SAN storage.
• The SAN requires the core switches be functional before it comes online.

As you can see, the dependencies quickly get complex.  If you have to shutdown everything, what order do you do it in?  How do you bring it back up?  I’ll be documenting all of this over the coming weeks and hopefully actually doing a test at some point.  More updates to come.

I must say I’m a but ashamed of this pictures. This rack is the core of our network. What you’re looking at is all the connectivity to our servers and SAN, our internet connection and firewalls, and all of the point to point circuits connecting the remote sites. Fortunately, it’s better than it used to be. I had the opportunity to do a lot of cleanup back in December when we installed the SAN, which also involved replacing our core switch.

As you can see a lot of progress has already been made. Unfortunately, I can’t simply rip it all apart and start over like I would with an IDF closet. All of our connectivity to EVERYTHING goes through this rack, so it had to be done in small steps late in the evenings or on weekends. The goal is to have everything cabled with proper length cables all bundled neatly with proper cable management in place. Cables are also being color-coded so that we can identify cables that care general LAN traffic from SAN, voice, Wireless and DMZ connectivity.