Las Solanas Consulting

Storage Virtualization | FAQs & Discussions

How To Build An Ultra-High Performance H/A Fibre Channel SAN using Dell PowerEdge Servers

In this example, we will build a turbo-charged Fibre Channel / iSCSI storage array or "SAN" using Dell servers. The objective is to achieve I/O performance equivalent to the so-called "Tier 1" Storage Frames, combined with High Availability — but at a fraction of the price of a Tier-1 solution. The choice of the Dell PowerEdge is completely arbitrary; we could build this same type of SAN storage array using servers from IBM, HP, Sun, etc.

The server was configured using Dell's online store, with an objective of only selecting components available there. Date of this configuration: October 21, 2006.

First off, we are going to use DataCore's SANmelody™ software to implement our SAN. This Storage Virtualization software offers many features associated with High-End storage systems. Two of these will be of interest to us here:

  • High-Performance through caching and an advanced real-time I/O subsystem
  • High-Availability through Synchronous Data Mirroring

We need to choose a PowerEdge server suitable to run the Storage Virtualization software that will power the SAN. Normally, any of their servers would qualify to run the DataCore software, but we are looking to achieve highest performance. To that end, I've decided on a PE6850, principally because of its PCI slots: four PCI Express™ and three PCI-X slots. Those slots will allow us to construct a SAN Storage Array like nothing available on the market today: a monster built for speed.

We will load the four PCI Express slots with QLogic QLE2462 4Gbit dual-port Fibre Channel HBAs. These cards are fast, rated at 150,000 IOPS per port. The SANmelody software will drive these ports as Fibre Channel "targets", effectively making them the front-end ports of our storage array.

As we are on the topic of front-end ports of a storage processor, it is worth noting the 6850 has two Gig-E Ethernet ports integrated. The SANmelody storage array software supports both FC and iSCSI, implementing iSCSI targets on any available Ethernet ports we choose. Although our objective is high-performance, it's great to have two iSCSI ports available for free.

One of the principle components of a Tier-1 storage array is its massive (and costly) cache. The PowerEdge 6850 supports up to 64GB of RAM. That's far more than we will need, but as one of our objectives is extreme performance, we will dope the 6850 with 8GB of RAM. SANmelody will allocate most of this memory for its sophisticated caching engine.

As far as processors go, we will be satisfied with two dual-core processors, choosing a reasonably fast config. CPU speed is not as important to us as PCI bus performance, FC channel performance, and cache. With SANmelody, the CPUs will be busy polling I/O completions and moving data between I/O ports and memory... not a massive amount of number crunching involved.

Let us turn our attention to the storage resources our SAN will use: disks, HBAs, and RAID controllers. The 6850 has an integrated dual-channel Ultra320 controller and 5 disk slots capable of holding 300GB Ultra320 disks, giving us 1.5TB raw in a 4U storage processor.

The remaining three PCI-X slots can be used to attach additional storage enclosures, as needed. Any mix of drive enclosure could be used to connect U320, FC, SAS, or SATA drives. As we're interested in performance, we could choose 4Gb FC drives, but in the interest of controlling costs while maintaining excellent performance results we will use the Dell PowerVault 220S. We can connect two of these per PCI-X slot using the dual-channel PERC 4/DC HBA. The 220 can hold up to 14 x 300GB Ultra320 drives, yielding 4.2 TB per enclosure, for a total of 25.2TB of raw storage over 6 enclosures, in addition to the internal 1.5TB on the 6850.

Obviously, our choice of drive influences performance (146GB 15K RPM vs. 300GB 10K RPM; or SAS vs. U320), and so will our choice of RAID (individual disk, RAID-1, RAID-5, RAID-10, etc.), but these choices are also determined by our capacity requirements as well as the nature of our I/Os (mix of Reads/Writes, average size of requests, random or sequential character of I/O). All of this also largely influences (and is influenced by) the role of our cache engine, which in many cases can mask performance inadequacies of the spindles. The Ultra320 solution is a safe choice, offering an excellent balance between performance and economy.

Here is the PowerEdge configuration:

PowerEdge 6850 Qty 1  
NEW 3.0GHz/800Mhz/4mb Cache, Dual-Core Intel® Xeon 7120M Processor, Windows Server® 2003 R2, Enterprise Edition, Includes 25 CALs Unit Price $21,610.00  
PowerEdge 6850
Catalog Number: 5 PE6850PAD
Module Description
PowerEdge 6850 NEW 3.0GHz/800Mhz/4mb Cache, Dual-Core Intel® Xeon 7120M Processor
Additional Processor NEW Dual 3.0GHz/800Mhz/4mb Cache, Dual-Core Intel® Xeon 7120M Procssrs
Operating System Windows Server® 2003 R2, Enterprise Edition, Includes 25 CALs
Memory 8GB DDR2 400MHZ(8X1GB), Single Ranked DIMMs
1st PCI Express QLogic QLE2462, Dual Channel, 4 Gbps, Optical, Fibre Channel HBA, PCI-E
2nd PCI Express QLogic QLE2462, Dual Channel, 4 Gbps, Optical, Fibre Channel HBA, PCI-E
3rd PCI Express QLogic QLE2462, Dual Channel, 4 Gbps, Optical, Fibre Channel HBA, PCI-E
4th PCI Express QLogic QLE2462, Dual Channel, 4 Gbps, Optical, Fibre Channel HBA, PCI-E
Primary Controller Embedded RAID - PERC4 Embedded Integrated
Hard Drive Backplane 1x5 SCSI Backplane, Hot Plug, SCSI Hard Drives only, PowerEdge 6850
Hard Drive Configuration Drives attached to embedded PERC4e/Di, RAID 5, Min of 3 drives required
Primary Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
2nd Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
3rd Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
4th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
5th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
Floppy Drive No Floppy Drive
Mouse No Mouse Option
Network Adapter Dual Onboard NICs
CD/DVD Drive 24X IDE CD-RW/DVD ROM Drive for PowerEdge Servers, All OS
Bezel Active ID Bezel Option
Chassis Configuration Rapid Rails for Dell Rack
Power Supply Dual 200 - 240 Volt Power Supply
Documentation Electronic Documentation and OpenManage CD Kit
Hardware Support Services 3Yr SILVER Support, 4Hr Onsite, S/W Support
Installation Support Services No Installation Assessment

Turning The Server Into A SAN Storage Array

So we've configured our high-performance SAN storage processor hardware with 8GB of cache, 8 * 4Gb FC ports, and we've even given it 1.5 TB of raw storage, all in a 4U enclosure. The one thing that's missing is the intelligence to turn this ordinary high-power server into a SAN: the SAN volume, caching, and LUN masking management, as well as the drivers to run the Fibre Channel HBAs in "target" mode so they'll behave like a storage array's Fibre Channel ports.

For this functionality we will use a unique software package from DataCore Software Corporation called SANmelody™. This software will turn the Dell PowerEdge 6850 into a SAN storage array far more powerful and performant than any Dell rebadged CX3 80 — but for less money.

Implementing Extreme High-Availability

To address our High Availability requirement, we turn again to the DataCore SANmelody software. The traditional midrange storage array achieves High-Availability through redundancy: two storage processors share a common backplane over which they can mirror their data (i.e. Mirrored Write Caching). Most of the tradition arrays implement mirrored write caching over a traditional media, such as a shared FC bus on the backplane. Both storage processors are attached to the physical spindles in a "dual-ported" fashion. While at first this appears to be an excellent means of protecting the data and keeping it available, the storage array itself is a single point of failure. I have heard numerous accounts first-hand from customers who have had downtime on their highly-available systems. A few have involved broken water pipes flooding an inadequately insulated data center. One involved a failed fibre channel drive in a StorageTek-badged array. The drive's failed behavior caused a "LIP Storm"; both storage processors to begin taking down RAID groups.

Such interruptions to production could be avoided by simply separating the dual storage processors, each with its own physical storage attached. Mirrored write caching could be implemented by a Fibre Channel inter-connect. This is exactly what we propose to do using the SANmelody software. The SANmelody software has an optional "Auto Failover" feature that creates a partnership between two SANmelody servers. For any given virtual volume, a synchronously mirrored pair can be created, effectively a RAID-1 mirror. If any component fails (cable, HBA, SANmelody server, etc.), the application servers can fail over to the surviving SANmelody node and continue to access their data. The SANmelody failover feature is an intelligent implementation with similar characteristics one would find on a mid-range or Tier-1 system. However, SANmelody takes High-Availability to the extreme, separating the storage processors and physical spindles.

SANmelody's synchronous data mirroring is implemented on a virtual volume basis. This is to say that you are not required to mirror all of the storage -- you only mirror those volumes for which you want extreme availability.

Of course, two SANmelody nodes also means double the number of front end ports, double the processing power.

We will create a second storage array using another, similarly configured PowerEdge 6850. We will separate the two servers, either across the data center, across the building, or across campus, depending on the availability of real estate and fibre access. We will interconnect the two SANmelody storage arrays using two FC cables, attached "point to point" between ports on two of our QLogic cards. Using two interconnects and a port from two of our FC HBA's will give us resiliancy against failure of a GBIC, HBA, or cable on the mirror. We've used two ports on each of the SANmelody servers, leaving us 6 * 4Gb ports per storage processor, or a total of 12 front-end 4Gb Fibre Channel ports. And don't forget, we also have 4 iSCSI ports available, should we need them.

Total cost of this Extreme Highly-Available, High-Performance two-headed monster? The two 6850 servers price retail at about $44K. The software required to implement the SAN will come in well below $20K, including 24/7/365 support. Total cost around $65K. This configuration will leave the CX3 80 in the dust — indeed, this configuration will perform as a Tier-1 SAN storage array.

Storage Expansion

To help us get an idea of the real cost of this solution, let's do the capacity expansion and fully populate our SAN storage processor with S220 trays and Ultra320 drives. To begin with, we'll need to add RAID controllers to our servers in order to attach the S220 trays. We have 3 available PCI-X slots on our PowerEdge 6850 in which we will install Dell's PERC4/DC dual port RAID controller cards.

The 220S supports both joined-bus and split-bus modes. The choice is yours, but in this configuration I will choose joined-bus in order to maximize the use of my PERC4/DC channels — with joined-bus mode I can attach 6 S220 trays to each of my PowerEdge SAN Servers. Here's the config details for the RAID controllers and S220 trays.

3 x PERC4/DC 2 Channel RAID Controllers Unit Price   $2397.00
Module Description
RAID Controller PERC4/DC, 2 External Channels, 0 Internal Channels
RAID Controller PERC4/DC, 2 External Channels, 0 Internal Channels
RAID Controller PERC4/DC, 2 External Channels, 0 Internal Channels

PowerVault® 220S External SCSI Storage Array, Rack Unit Price   $14,309.00
PowerVault S220
Catalog Number:  5 PV220SCSI221
Module Description
PowerVault 220/221 PowerVault® 220S External SCSI Storage Array, Rack
Primary Controller PowerVault 22XS, Single, U320 Enclosure Management Module
Power Supplies Redundant 600W Power Supply for PowerVault 22XS
Locking Bezel PowerVault 220 Locking Bezel
Rack Rails Rapid Rails for Rack Mounting the PowerVault 220S in Dell Racks
Cables Two 4M Cable, PV22XS to PERC2/PERC3/PERC4/PERC4e/39160(Wide HD/Slim VHDCI)
1st Hard Drive 300GB, U320, SCSI, 10K RPM, 80 Pin Hard Drive for PowerVault
2nd Hard Drive 300GB, U320, SCSI, 10K RPM, 80 Pin Hard Drive for PowerVault
3rd Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
4th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
5th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
6th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
7th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
8th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
9th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
10th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
11th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
12th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
13th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
14th Hard Drive 300GB 10K RPM Ultra 320 SCSI Hard Drive
Hardware Support Services SILVER, 3YR Same Day Onsite Svc(24 Hrs-7 Days)+L2 Suppt+Adv SW Suppt (3Res,
Installation Services PowerVault Installation Declined

So we can see that a loaded S220 with attached PERC4 RAID HBA will run us about $15K (NB: each PERC4 can attach two S220 trays). For $15K we can easily add 4.2TB raw to our SAN; $30K if we implement synchronous mirroring for High Availability.

To fully populate our SAN by adding the three PERC4 RAID controllers and the 6 S220 trays will cost us $88251.00 per server or $176,502 in an extreme high availability configuration with synchronous data mirroring. Total cost for 53.4TB raw (26.7TB all mirrored) high-performance SAN storage, including support: less than $240,000. A very interesting value proposition: higher performance, much lower cost than anything the "3 letter acronym" storage vendors can propose.

Las Solanas Consulting is not a DataCore or Dell reseller, and DataCore does not publish their prices on the web. Please contact DataCore™ or a DataCore reseller for pricing information.

If you want to take a test drive of my recommended SANmelody configuration, contact Las Solanas Consulting or DataCore Software Corporation directly. Optionaly, you can download a free, no-obligation 30-day evaluation. The evaluation software includes iSCSI and FC support, as well as support for their unique Thin Provisioning feature — however, it does not have support for the Synchronous Mirroring feature. Nonetheless, you'll see for yourself how SANmelody out-performs the traditional storage vendor's SANs or Storage Arrays.