Las Solanas Consulting

Storage Virtualization | FAQs & Discussions

How To Build A High Performance H/A iSCSI SAN using IBM System x3655 Servers

In this example, we will build a scaleable iSCSI SAN storage array featuring Synchronous Data Mirroring for Extreme High Availability using IBM System x (a.k.a. xSeries) servers.

First of all, we will need some mechanism to turn an x86 Server into a SAN Storage Array. To that end, we will use a unique software package from DataCore Software Corporation called SANmelody™. This Storage Virtualization software has the logic needed to implement a SAN Storage Processor and offers many features associated with High-End storage systems. Two of these will be of interest to us here:

  • High-Performance through caching and an advanced real-time I/O subsystem
  • High-Availability through Synchronous Data Mirroring over iSCSI and/or Fibre Channel

Building The Storage Processor Hardware

The choice of the IBM System x3655 for this example was based on the server's expansion slots and internal drive slots. The x3655 comes in flavours of either eight 2.5" or six 3.5" internal drives. The 3.5" slots are hot swappable and can accommodate the large 500GB SATA drives. The server offers 4 expansion slots (2 PCI Express x8, 1 PCI Express x4, and a PCI-X 133 MHz) allowing us to add in additional RAID controllers, Gig-E and / or FC cards.

We begin by selecting processor and memory options. We decide on a good price/performance balance and choose an AMD Dual-Core Opteron 2.4GHz and add in 4GB of RAM. The server's memory will be used as storage processor cache.

Next we choose the disks. The x3655 offers a choice between SAS and SATA drives. In the interest of maximizing capacity, we will load all 6 3.5" Hot-Swap drive slots with 500GB SATA drives. While not strictly necessary to implement data protection in this HA configuration, we will optionally upgrade the internal storage controller to a ServeRAID-8K, for implementing RAID-5 on the internal drives. More on this later.

The x3655 has two integrated Gig-E ports which can serve as iSCSI targets. We will add two additional Gig-E HBA's, bringing our iSCSI port count to 4.

Finally, we add in an optional redundant power supply. I would recommend adding a UPS, as well.

The following IBM System x3655 server was configured using IBM's online store. Date of this configuration: November 12, 2006.

IBM x3655 Server

Description Part number Unit $ Qty Xtnd $
IBM System x3655 7985AC1 $8,070.00 1 $8,070.00
System Base 2267 $1,545.00 1 $1,545.00
Initial processor: Dual Core Opteron Processor Model 2216 (2.4GHz 2x1MB L2 95w) 1269 $735.00 1 $735.00
Memory: 1.0 GB DDR 2 667MHz DIMM memory 0598 $229.00 4 $916.00
Storage subsystem: 3.5" SATA Hot-Swap Enabled System 2468 1
Configuration option: Internal RAID - Cabled only, Setup by Customer 9011 1
Storage adapter: IBM ServeRAID-8k SAS Controller 1670 $359.00 1 $359.00
Hard disk drive: 500GB 7200 rpm 3.5" Hot-Swap SATA HDD 5196 $619.00 6 $3,714.00
Optical drive: CD-RW/DVD Combo V Ultrabay Enhanced 4144 $159.00 1 $159.00
I/O riser cards: HTX RISER CARD 2081 $59.00 1 $59.00
Additional PCI Express adapters: NetXtreme 1000 Express G Ethernet Adapter 1486 $169.00 2 $338.00
Power supply: Redundant Power Supply 835W (Standard) PWRS003 1
Additional power supply: xSeries 835W Redundant Power Option 1968 $239.00 1 $239.00
Line cords: Rack power cable - 2.8m, 100-240V, C13 to IEC 320-C14 (WW) 6311 $3.00 2 $6.00
Language pack: System Documentation and Software-US English 8424 1
BOXLINE FLASHABLE CODE 2085 1
System Planar 1127 1
IBM Blank Tape Filler 2052 1
3.5" SAS Bay support kit 0977 1
Cooling Fan Asm. 0979 5
Entry Cable Management Arm 2005 1
Cooling Fan (Standard) SCF0001 5
System Packaging - WW 0787 1
No Preload Specify 9206 1
Dummy Heatsink 0963 1
2U Bracket for 39Y6101 2034 1
Rack Kit 0975 1

Total cost of the server as configured: $8070.00.

Turning The Server Into A SAN Storage Array

So we've configured our high-performance SAN storage processor hardware with 4GB of cache, 4 Gig-E ports, and 3.0TB of raw storage, all in a 2U enclosure. The one thing that's missing is the logic to turn this standard high-power server into a SAN: the SAN volume, caching, and LUN masking management, as well as the drivers to run the Gig-E ports as front-end iSCSI "targets".

For this functionality we will use a unique software package from DataCore Software Corporation called SANmelody™. This software will turn the IBM System x3655 into a SAN storage array far more powerful and performant than any storage appliance — but for less money.

SANmelody™ runs on a standard, familiar Windows OS platform. We will first perform a clean install of Windows OS (such as XP Pro SP2 or 2003 R2) on the x3655 server. Then we will install the SANmelody software. Upon rebooting the machine, the SANmelody software will load its target drivers on any available ports (Gig-E for iSCSI, FC on any Fibre Channel HBAs) and once the SANmelody services are started, the server will behave as a storage processor. The SANmelody software will drive the Windows OS as a real-time process, implementing I/O Polling on the iSCSI and FC ports.

The SANmelody software is managed from the MMC (Microsoft Management Console), under the storage section. The interface is intuitive and an online help system provides a "Quick Start" tutorial to get us going. If you can manage a Windows server, you can manage a SANmelody™ SAN.

Thus far, we've built a scaleable iSCSI SAN storage array for very little money. This single storage processor model features 4 iSCSI front end ports, 4GB of cache, and 3TB of SATA storage. This iSCSI SAN array still has two slots available so we could add an external RAID controller to increase our capacity, as well as add additional Gig-E ports or even a Fibre Channel HBA. This SAN will cost significantly less than, say, an EqualLogic PS50E (limited to only 3 iSCSI front end ports, 1GB of cache and 1.7TB of SATA storage), but will largely outperform it.

Still, like the PS50E and other similar appliances, our Single Storage Processor SAN is a Single Point of Failure (SPOF). To address our High Availability requirement, we turn again to the DataCore SANmelody software.

Implementing Extreme High-Availability

The traditional midrange storage array achieves High-Availability through redundancy: two storage processors share a common backplane over which they can mirror their data (i.e. Mirrored Write Caching). Most of the traditional arrays implement mirrored write caching over a traditional media, such as a shared FC bus on the backplane. Both storage processors are attached to the physical spindles in a "dual-ported" fashion. While at first this appears to be an excellent means of protecting the data and keeping it available, the storage array itself is a single point of failure. I have heard numerous accounts first-hand from customers who have had downtime on their highly-available systems. A few have involved broken water pipes flooding an inadequately insulated data center. One involved a failed fibre channel drive in a StorageTek-badged array. The dual-ported drive's failed behavior caused a "LIP Storm"; both storage processors to begin taking down RAID groups.

Such interruptions to production could be avoided by simply separating the dual storage processors, each with its own physical storage attached. Mirrored write caching could be implemented by a Fibre Channel or iSCSI inter-connect. This is exactly what we propose to do using the SANmelody software. The SANmelody software has an optional "Auto Failover" feature that creates a partnership between two SANmelody servers. For any given virtual volume, a synchronously mirrored pair can be created, effectively a RAID-1 mirror. If any component fails (cable, HBA, SANmelody server, etc.), the application servers can fail over seamlessly to the surviving SANmelody node and continue to access their data. The SANmelody failover feature is an intelligent implementation with similar characteristics one would find on a mid-range or Tier-1 system. However, SANmelody takes High-Availability to the extreme, separating the storage processors and physical spindles.

SANmelody's synchronous data mirroring is implemented on a virtual volume basis. This is to say that you are not required to mirror all of the storage -- you only mirror those volumes for which you want extreme availability.

Of course, two SANmelody nodes also means an increase in the number of front end ports and double the processing power and spindles. You can load balance by distributing the volumes between the two processors.

We will create a second storage array using another, similarly configured IBM System x3655. We will separate the two servers, either across the data center, across the building, or across campus, depending on the availability of real estate and fibre access. SANmelody can use both iSCSI and FC to implement mirror channels for the Synchronous Data Mirroring. Rather than invest in FC technology, we will use iSCSI for the mirroring. Although we could interconnect the two SANmelody storage arrays over our IP SAN, it is preferable to isolate the mirrored write traffic, so I propose we use a crossover cable between two of our Gig-E ports. Thus, we've used one iSCSI port on each of the SANmelody servers for the mirror traffic, leaving us 3 front end iSCSI ports per storage processor, or a total of 6 front-end iSCSI ports. And don't forget, we still have 2 expansion slots available should we need them.

Advantages Of This System Over A Similarly Priced EqualLogic PS50E

Taking the SANmelody™ software approach to developing a SAN virtual infrastructure has numerous advantages over the appliance approach. Comparing, for example, the similarly priced low-end EqualLogic PS50E appliance with our High Availability x3655 / SANmelody™ configuration, several benefits are evident.

This last point is particularly important. Although the PS50E and our SANmelody™ H/A configuration have similar price points, the EqualLogic appliance features "built-in obsolescence"; SANmelody™ software will keep its value long after the PS50E is end-of-life. As new features are introduced in SANmelody™, support-paying customers will get the upgrades for free.

A few years down the road as server platforms continue to evolve, and as the x3655 has come to end-of-life, we can transfer our SANmelody™ software to a newer, more powerful server. As new disk technologies come to market, our SANmelody™ server will be ready to take advantage of them. With SANmelody's Synchronous Data Mirroring, you can even plan the migration to the new server or technologies without downtime.

On the other hand, the appliance approach limits you to the feature set and technologies available at the time of purchase. An EqualLogic appliance you purchase today won't be able to adapt to use new technologies as they become available. And when the first three years of support have run out and you get the support renewal bill, it will be time to forklift the PS50E and replace it with something else. You'll have to buy the entire feature set all over again.

Las Solanas Consulting offers this information as a service and is not a DataCore or IBM reseller. DataCore does not publish their prices on the web. If you would like more information, please contact DataCore™ or a DataCore reseller for pricing information.

If you want to take a test drive of the SANmelody software, you can download a free, no-obligation 30-day evaluation. The evaluation software includes iSCSI and FC support, as well as support for their unique Thin Provisioning feature. You'll see for yourself how SANmelody out-performs the traditional storage vendor's SANs or Storage Arrays.