Las Solanas Consulting

Storage Virtualization | FAQs & Discussions

True Business Continuity: High Performance, Self-Healing Storage Goes Virtual

A White Paper by Tim Warden

May 1, 2010

DataCore Xiotech

Back in 2008, I was invited to work a technology event with Thin Client Computing, a Scottsdale, Arizona consulting firm renowned for their expertise in Citrix and all things virtual. If you ever get an invitation to a Thin Client Computing event, I highly recommend you go. Steve Greenberg is the consumate host and always chooses a classy venue for his events, replete with good grub, drinks and entertainment. This particular event was held at Boulder House, which is well worth the visit. No pushy sales, just an opportunity to inform yourself and have a good time.

Following the lunch presentation, the event was a kind of "chat room", only not in the virtual sense. Each vendor was given space at Boulder House and the guests could meander from booth to booth and learn about the different technologies presented. Of course, I was representing DataCore Software, but Citrix, Xiotech and a few other vendors were also present.

I met the SE from Xiotech, a really great guy named Mike Chaput, and we began discussing our respective products. I think Mike and Steve had it figured out long before I did, but we all drew the same conclusion: our two products were very complementary and represented a radical departure from the boring traditional approach to storage touted by most of the other vendors.

In short, the Xiotech Emprise 5000 is a high-performance, self-healing storage "brick". With the Emprise 5000, Xiotech has differentiated themselves from the rest of the hardware SAN vendors by focusing on the three principal areas a hardware vendor should focus on: Reliability, Performance, Ecology. As for the advanced SAN features, DataCore SANmelody is storage virtualization software that turns a Windows server into a dedicated, high-performance SAN controller with a rich feature set. SANmelody can virtualize any block level disk storage you can discover in the Windows device manager... and as such it can virtualize the Emprise 5000's logical disks.

"Pairing SANmelody and the Emprise 5000 is like, 'You got peanut butter in my chocolate'... or is it 'chocolate in my peanut butter'?"

—Steve Greenberg, Thin Client Computing

A REVOLUTION OF THE SPINDLE

The Xiotech Emprise 5000 is based on their ISE or "Intelligent Storage Element" technology which they acquired from partner and investor Seagate Technology. This may sound like the typical vendor's marketing spin, but the ISE truly is a revolution in disk technology. Every aspect of this shelf has been designed for reliability and performance, using techniques you will find on no other SAN storage platform. From its self-healing drives and its innovations for radically reducing heat and vibration (the primary disk killers), right down to the layout of data across the platters, the ISE is a significant advance in physical storage.

Front Bezel of Xiotech Emprise 5000
Xiotech Emprise 5000, Front Bezel Removed

RELIABILITY

As it turns out, the vast majority of failed disk drives returned to the storage vendors are, in reality, perfectly healthy disks. Seriously. Most storage controllers don't bother to analyze the errors returned by the disks, mostly because they wouldn't be able to act on the analysis anyway... so they simply mark the disk as failed and go into a degraded mode and start rebuilding the RAID stripe using the hot spare if one is available.

But as my friends at Xiotech have explained, oft times the disk is at worst only in need of a re-calibration and low-level reformat. The storage admin will yank the failed drive from the enclosure and RMA it to the vendor, plugging in one of their spares. The vendor in turn sends the failed drive back to the manufacturer (Seagate or HDS) who will put the drive on the bench for analysis. Obviously the vendors have diagnostic tools that can access the drive's telemetry and determine if there is a problem. In over 90% of cases, their is nothing at all wrong with the drive. They command the drive's internal firmware to "remanufacture" 1 itself, slap a "refurbished" sticker on it and send it back out to the vendors for use as a spare.

These diagnostic utilities and the access to firmware commands to remanufacture a drive are reserved to the manufacturers' bench, but via an exclusive agreement with Seagate, Xiotech's advanced storage controllers (or MRC's) are capable of analyzing and automatically migrating block data from a trouble drive and remanufacturing the drive on the fly... without user intervention.

Xiotech Emprise 5000 MRC's
Dual Active/Active Controllers (MRC's)

But what causes the drives to "fail" in the first place? In most cases the answer is simply vibration and heat. As vibration of the disks increases, the heads are more likely to have read/write errors and have to "retry". The retry means additional rotational latency as we wait for the targetted block to spin back under the head. The drive will only attempt the operation so many times before giving up and returning an I/O error. And as we've said, most storage arrays may tolerate a certain number of those errors, retrying the I/O, but eventually they'll deem the drive bad. If you've ever had an opportunity to hold a drive when it's powered on, then you know that indeed whether 5400 RPM or 15K, those drives do vibrate and yes they get hot, but of course they're designed to function within certain tolerances commonly found in real life applications.

Think about the typical drive shelf in a storage array... 12 or 14 drives vertically seated in line, all rotating in the same direction... You don't have to be a physics major to realize that the harmonics created by the cumulative vibration of the juxtaposed drives only exacerbates the vibration problem. According to Xiotech, it is not uncommon for an I/O opperation on a particular drive to fail because of excessive vibration of a neighboring disk. One can imagine that excessive heat generated by the drives can add to the problem of disk heads straying just far enough out of tolerance to cause excessive read / write errors. If you could reduce heat and vibration, you could certainly reduce those errors and prolong the life of the drives. And that's another of the innovations Xiotech has effected in the Emprise 5000.

"A solution so simple, so obvious... and yet it only took the industry 30 years to figure it out!"

—Roger Kelley, Xiotech

Whereas the traditional drive shelf seats the drives inline in removable disk trays, the Emprise 5000 mounts the disks in a Magnesium alloy DataPac. The DataPac contains 10 LFF or 20 SFF drives, all counter-mounted in such a way that their rotational vibrations cancel out. Whereas the traditional SAN shelf in your data center measures vibration in 20-30 RADs, the Emprise 5000 has reduced vibration to 2-3 RADs... a radical reduction in vibration. As Xiotech Principal Architect Roger Kelley puts it, "A solution so simple, so obvious... and yet it only took the industry 30 years to figure it out!"

View of Mag-Alloy Frame of DataPac
View of Mag-Alloy Frame, Back of DataPac

The design of the magnesium frame assures the maximum amount of airflow over the drives, improving heat dissipation. The test unit I have here in my lab generates less heat than most of my other servers, and the fans often spin down to a soft whine amusingly reminiscent of a couple of kitten's crying... far less annoying than the jet engine rush of another vendor's SAN array in same said lab. Speaking of fans, Xiotech even designed the fans to last using the same quality of ball bearings used by Seagate on the drive spindles.

Another important advantage to the self-healing enclosed DataPac is the elimination of human intervention (and thus human error) in replacing drives. I suppose you've never had someone on your team pull the wrong drive by accident? Worse still, shove that drive back in, and then immediately pull the right failed drive? But now your RAID group is compromised and the volume or volumes taken offline... And sadly, this is not that uncommon an occurrence. With the Xiotech self-healing DataPacs, there are no failed disks to pull, no badges on the table.

Xiotech Emprise 5000 DataPac
DataPacs come in 10 LFF Disk and 20 SFF Disk Configurations

PERFORMANCE

So far we've discussed the revolutionary ways the Emprise radically improves the reliability of the storage. Now let's talk about performance. Again, Xiotech has applied their advanced knowledge of drive internals to make the Emprise 5000 apples-to-apples the fastest performer on the storage vendor block. Xiotech has published their SPC-1 and SPC-2 audited results with Storage Performance Council so you can see for yourself the Emprise delivers the highest IOPS per spindle and dollar of any hardware SAN storage array in the industry.

So how do they do it? Whereas the traditional storage vendors build their RAID stripe across entire disk drives, Xiotech uses their patented RAGS (Redundancy Allocation Grid System) technology to stripe the DataPac's drives right down to the individual platter. If you've ever disected a drive, you've discovered there are typically four platters on the spindle, thus 8 oxide surfaces, with an actuator & head for each surface. As most SAN vendors don't have visibility at that level, the best they can do is offer you a means of optimizing performance by "short stroking" the drives -- which is to say not filling up the drives so that the heads will stay on the outer tracks of the spindle where performance is highest, all the while keeping seek latency to a minimum. With Xiotech's RAGS technology, the stripe is constructed in such a way that you get optimised, consistent performance regardless of how fully you've provisioned the DataPac. As you carve out LUNs on the DataPac, you choose from RAID1 and RAID5 which will be allocated from the RAGS stripe.

Where most vendors address reliability by using RAID6 (thus decreasing performance with two drives worth of parity) and hot spares (again, idle spindles not contributing to performance), the Emprise 5000 interleaves two disk's worth of spare space across the RAGS stripe, so that all spindles in the DataPac contribute to performance. With the SFF DataPacs, that's 40 drive's worth of performance in a 3U enclosure.

Of course, as we discussed earlier, the unique way Xiotech has reduced disk vibration below 3 Rads means that there will be far fewer failed initial read/write attempts, thus far fewer retries with the corelated reduction in rotational latency per failed attempt.

Finally, if the Xiotech controllers determine that certain blocks or sectors are indeed faulty, they will simply address the failure at the sector level, using the RAID parity information to rebuild the sector — not the entire disk — on reserved spare space. With larger capacity drives on traditional storage controllers, the rebuild of an entire disk can take a long time, seriously degrading performance and making the stripe's data vulnerable to a second disk failure.

Back View of Xiotech Emprise 5000
Back View of Emprise 5000

I won't go into detail on the numerous other advanced technologies employed in the ISE, but suffice it to say this brick is built to last and delivers outstanding performance and value for the money. Xiotech is so sure of this technology they sell the Emprise 5000 hardware with a full 5 year warranty — with no annual maintenance and support contract. Revolutionary.

DataCore SANmelody

ISE-ING THE CAKE

On its own, the Emprise 5000 is an excellent value: a highly-available, high capacity, high performance Fibre Channel array. The device is easy to setup and manage via an intuitive web-based GUI.

Adding SANmelody as a virtual storage controller will give the Emprise a complete Tier-1 feature set. If you're familiar with this website you already know that SANmelody is a software package from DataCore that turns standard x86/x64 servers into advanced, feature-rich SAN storage controllers.

SANmelody leverages the open x86/x64 server architecture and ubiquitous Microsoft Windows platform. The software employs the standard server RAM as storage processor cache. NICs are used for iSCSI targets whereas common Fibre Channel HBAs (such as those from Qlogic and Emulex) are used to implement FC targets.

With SANmelody, you can take advantage of new technologies as they come available. For instance, in February, 2008 when QLogic and Emulex shipped their 8Gb FC HBA's, SANmelody and her big sister SANsymphony became the first 8Gb FC SANs on the market. The DataCore target & initiator drivers were compatible with the new cards. The rest of the industry would wait for the traditional hardware vendors to catch up. As for 10Gb iSCSI? DataCore has supported that since day one... their iSCSI target drivers attach to the IP stacks of the host on which the software is running. If that host has 10Gb or even teamed NICs, SANmelody will be able to use them as iSCSI targets.

Funny that EMC, which also uses x86 boards and Windows (XP Embedded in the CX3; 2003 Storage Server Edition in the CX4) in a proprietary system, only introduced 8Gb FC many months later in the CX4 and, in their datasheet for the CX4 cites future proofing your investment as "lets customers add future connectivity like 8 Gb/s FC and 10 Gb/s iSCSI...".

With DataCore, that future is now. Or yesterday... over two years ago, as it were. As for new disk technologies, SANmelody is ever-ready. As it runs on a Windows platform, practically any new disk technology available for Windows can be used by SANmelody. Solid State Disks? But, of course! And those are becoming an interesting possibility as the costs come down and availability goes up.2

If Windows can see and use the disk, so can SANmelody.

And that's where the Xiotech ISE technology comes in. Xiotech invested in making the hardware as reliable and high performance as possible. DataCore brings the advanced SAN feature set to the party. With SANmelody, the ISE gains features like:

  • Native 2, 4, 8GB FC connectivity
  • Native 1GB, 10Gb, and teamed iSCSI connectivity
  • Tiered Storage Pools
  • Thin-Provisioning
  • Pointer-based Snapshots and Full Volume Clones
  • Auto-Failover & Failback Synchronous Mirrors
  • Active/Active Stretch Clustering
  • Asynchronous Replication over Standard IP
  • Numerous Tiering and Migration Utilities

And because SANmelody is an open, Windows-based virtualized storage platform, you can easily integrate and inter-work with other disk technologies, such as other FC arrays or iSCSI appliances and direct attached shelves with common RAID controllers. If Windows can see and use the disk, so can SANmelody. This means you can use SANmelody to integrate Xiotech into your existing environment and repurpose your older SAN storage array... perhaps take it off maintenance and use it for your DR site.

MIND YOUR OWN BUSINESS CONTINUITY

Business Continuity is a term routinely used (and abused) to define a form of data and application availability that goes beyond the standard "High Availability" touted by most of the IT hardware vendors.

Consider if you will, an EMC CX3 or CX4 series SAN storage array. The array is delivered as a 19" rackable unit, complete with dual Storage Processors (or SP's) that plug into a common backplane, implementing mirrored write cache between the controllers. The controllers "dual-port" the attached disks in the DPE and optional DAE's. Add in dual power supplies, redundant cooling and battery backup (a glorified UPS) and what do you have? A highly available 19" rackable Single Point of Failure.

That's not an indictment of the traditional storage vendor. That's just reality. The DPE (Disk Processing Enclosure) is not immune to localized incidents such as rack power failures or fire sprinklers unfortunately active and located over the rack. And if you point this out, you won't get disagreement from the vendor; most of the vendors offer a "Business Continuity" solution for just that case.

The "Business Continuity" solution they will propose uses Synchronous Mirroring between two similar (same vendor, same model) SAN storage arrays, in which one highly available array copies every write in lock-step to the secondary array. The idea is to physically separate the two SAN arrays. You'll have two copies of the data: one locally in the datacenter, another across the street, campus or metropolitan area.

While you will have two copies of the data on physically separate disks, they are also on two logically separate disks. The primary SAN — already considered a highly-available device — presents the SAN volumes as multipath disk devices. The secondary SAN has a copy of the data, but will not present the copy without user intervention.

Although the SAN vendors can propose Professional Services for "Script and Flip" style automation, it is a costly and complicated venture involving scripts on your application servers to implement a failover... but not a failback.

It is interesting to note that the vendors sell their Business Continuity solution as a costly software add-on running on their SAN controller's hardware.

DataCore takes Business Continuity to the logical conclusion. Since the SAN storage controller is understood to be an environmental single point of failure, why not treat the two physically separate SANs as one logical unit? Create a "Storage Cluster" if you will. Take two SANmelody servers — each with their own directly attached storage — and partner them as if they were two storage processors (or SPs) of the same logical SAN. They implement synchronous mirroring with mirrored write caching over FC and/or iSCSI connections and they each actively present the LUNs. If one of the SANs fails or needs to be taken offline for maintenance, the other SAN continues running, picking up the load of its partner. The application servers simply fail any outstanding I/Os over to the running partner via multipathing and production continues without a hiccough. No user intervention required. No lengthy pauses waiting for failover to occur. And because they are autonomous enclosures, they can be physically separated across campus or across town to implement a true active/active "Stretch Cluster" or "Metro Cluster".

Achieved goal? No more storage-related downtime... planned or unplanned.

Combining Xiotech's self-healing ISE technology with SANmelody's multipath auto-failover synchronous stretch mirroring yields the ultimate in hassle-free business continuity and peace of mind for the storage admin.

Active/Active Stretch Cluster

SPINNING TOPS

This solution is fast... blistering fast. Combining Xiotech's legendary RAID virtualization and optimized controller code with the chart-topping performance characteristic of DataCore Software products, the SANmelody ISE package spins circles around the traditional SAN storage arrays on the market.

But don't just take my word for it. Xiotech has recently published their numbers on the Storage Performance Council's website (http://www.storageperformance.org).

KNIGHT TRANSPORTATION — A CASE STUDY

Shortly after our meeting at Boulder House, Mike brought me into a storage opportunity he had with Sanity Solutions at Knight Transportation's headquarters in Phoenix. Knight had been having some serious challenges with another vendor's "highly available" product that wasn't living up to the name. Hardware component failures, "failover" that didn't and accumulated downtime had left them pretty frustrated and suspicious of storage vendors' high availability claims. We met with Shawn Smith and Alex Trussler in their IT department.

It had never occurred to me how important IT could be to a freight trucking company. As Shawn explained, logisitics is everything and without their IT services, business pretty much comes to a screeching halt once a rolling truck gets to its destination. Their campus is west of Sky Harbor airport, and with two buildings separated about one kilometer they wanted to build a stretch cluster so if they had any issues at either of the buildings, production would continue uninterruped at the other building. They installed an impressive diesel generator to make sure even a general power outage wouldn't stop the trucks from rolling.

Everything was to be redundant from servers to network infrastructure to storage. They were in the process of implementing server virtualization with VMWare ESX. An active/active stretch SAN was the missing piece.

Over the course of a few weeks they asked a lot of questions in order to fully understand different failure scenarios and how the proposed solution would behave. I took them through the details and explained a few of the nuances of building a stretch cluster. By the way, I really appreciate those deep-dive questions — I'm sometimes amazed at how many customers will make a large capital expenditure based on emotion and half-baked explanations from some of the vendors. Alex and Shawn and their boss weren't going to make that mistake. Knight choose the SANmelody / ISE solution and as a DCIE (DataCore Certified Implementation Engineer) I performed the SANmelody installation and configured the storage on the two ISE units. That was January 2009.

I stopped in recently (March 2010) to see them and find out how things are going. They're very happy with their DataCore/Xiotech/VMWare environment and told me their business was growing and that they were continuing to grow their stretch cluster, adding additional VMWare ESX hosts.

Click here to read the Case Study on Knight Transportation.

SUM IT UP

Since the Knight installation, DataCore and Xiotech have been mutually installed in many accounts across the states. It's an elegant solution, the two products go together like hand in glove. The Emprise 5000 delivers ultra-reliable, high performance storage hardware; SANmelody gives it the rich feature set and as a controller head, provides the aggregation of multiple Xiotech units. Additionally, it allows customers to virtualize their existing SANs to unify and manage all their storage under one "pane of glass".

NEXT STEPS

Please contact DataCore™ or a DataCore reseller for pricing information.

DataCore only sells its solutions through value added resellers or solution integrators, and a few of DataCore's resellers also resell Xiotech. While this list is non-exhaustive, here are three with whom I've worked extensively on DataCore/Xiotech deployments and with whom I can wholeheartedly endorse. These three resellers (like the vast majority of my DataCore resellers) are "technical top-heavy", preferring to base their business around the technical expertise of their engineers instead of the smooth talk of some slick sales guys.

  • Thin Client Computing

    Scottsdale, Arizona

    Steve Greenberg of Thin Client Computing is the visionary who first realised that pairing DataCore and Xiotech was like, in his words, "you got peanut butter in my chocolate"... or is it "chocolate in my peanut butter"? Steve is well known in Citrix circles and although Thin Client's expertise is as broad as it is deep, if you're looking at a Citrix or virtualization project, I suggest you give Steve a call. Thin Client's list of reference customers is impressive with several enterprises of world reknown.

  • Sanity Solutions

    Denver, Colorado & Phoenix, Arizona

    President & CEO Jason Cherveny is a storage veteran, having worked for many years at Xiotech. Don't be deceived by the name; while Sanity does indeed specialize in all things storage, his engineers frequently consult with Sanity's customers on server virtualization, cloud computing, networking, business continuity, backup, disaster recovery, portable secure data centers, power management, etc.

  • Mirazon

    Louisville, Kentucky

    I met Barry Martin & Craig Stein several years ago at a VMworld show. Their knowledge of all things VMWare and Microsoft is impressive, they're good communicators and a pleasure to work with. The company name is derived from the Japanese word "miraizou" meaning "vision of the future". Barry and Craig exemplify that image; in their quest to offer best of breed technologies for their customers, they do extensive research and thoroughly test each new product they integrate into their line card. DataCore and Xiotech figure among those best of breed offerings.

ABOUT THE AUTHOR

Tim Warden (DCIE, VCP) has been in the IT industry for nearly 30 years, both in R&D and in sales. Tim has a Bachelor of Science in Computer Science and Mathematics from Furman University in Greenville, SC. While still fluent in several programming languages, don't ask him to solve any so-called "Ordinary" Differential Equations. He's programmed from punch cards to PHP but you'll never hear him talking registers, stacks or heaps at a beer bash. After 15 years in R&D at Apple Computer he'll tell you that if you cut him, he still bleeds 6 colors. During his tenure at Apple, he was afforded the opportunity to work in Paris in Apple's European R&D group, and subsequently spent 15 years living in France. Tim is fluent in French, both written and spoken.

While SCSI drives were peripheral to his job at Apple, storage became an important part of his career when he joined Data General around the dawn of Fibre Channel, a few years before DG was acquired by EMC. He had his epiphany about the enormous possibilities of storage control as software while DG was developing their Wintel-based Clariions. A few years later in 2002, he discovered DataCore Software whose SANsymphony product eclipsed the Clariion as an open Storage Virtualization platform, decoupled from proprietary hardware.

On the personal side, Tim is a native Tucsonan and loves the Sonoran desert, la douce France, good food and wine, photography (argentique & numérique), his stereo and old-school vinyl collection, and his lovely wife Martine, a beautiful lady from Dijon, France.

FOOTNOTES

1 - Every disk drive that comes off the assembly line — whether the 2TB SATA drive you just bought at Fry's or Best Buy or the $2000 15K FC drive EMC sold you — has a rudimentary firmware that allows the drive to "manufacture" itself. Plugged into the vendor's bench, commands can be issued to cause the drive to run diagnostics, recalibrate itself, do a low-level format in which it writes its tracks, etc. I'm over-simplifying the process simply because, as much as I'd like to think myself a storage guru, like most folks I'm not that privy to what happens under the warranty seal of a disk drive. If you'd like to know more, I can put you in touch with a couple of the real industry gurus at Xiotech who I'm sure can explain the details to both of us!

2 - I wrote this white paper during a few of the numerous, unglorious flights I make to/from my home town of Tucson... on my MacBook Pro which has a 256GB Solid State Drive. It rocks.