Las Solanas Consulting

Storage Virtualization | FAQs & Discussions

How to Implement Snapshots, Data Mirroring and Replication on Symmetrix and DMX storage without using BCVs, TimeFinder or RDF

A White Paper by Tim Warden

November 17, 2006

Despite their unconventional start as an office furniture supplier, EMC made its name with their Symmetrix (a.k.a. DMX) product line. These Tier-1 SAN storage servers offer a rich feature set including snapshots and data replication, not to mention performance tuning features. Unfortunately, those advanced features come at a significant price, both in licensing costs and capacity utilization.

EMC licenses each feature across the entire storage park. If you have 4 loaded DMX servers in your data-center and you want to implement TimeFinder for a few of the volumes on one of the arrays, you have to buy the license for the whole installation... plus annual support. It's not uncommon to hear of customers spending over $2 million to implement TimeFinder and RDF in a large data-center. A very costly undertaking.

But the costs are not limited to licensing and support charges.

DMX storage admins are only too familiar with the dreaded BCV or Business Continuance Volume, a ravenous monster that consumes mass quantities of your expensive Tier-1 storage in order to implement the advanced features of TimeFinder and RDF. Like ordinary volumes, BCV volumes are based on the meta devices of the DMX. If you wish to make a TimeFinder copy of a particular volume, you must create a BCV volume of equal size. If you wish to make TimeFinder copies of all your volumes, you'll need to allocate half your logical disk space to accommodate all those BCVs. If your meta volumes are based on RAID-S group hypers, you pay an additional penalty in parity disk space. In the extreme case that all your meta volumes are based on RAID-1 mirrored hypers... only 1/4 of your Tier-1 storage is effectively usable space.

RDF or SRDF further complicates matters. The data replication features requires a compatible Symmetrix or DMX, and of course BCVs are used on both source and destination Symms to implement the replication.

Storage Virtualization To The Rescue

As it turns out, two of the advanced features of the Symmetrix / DMX can be found in Storage Virtualization products such as SANsymphony from DataCore Software Corporation. SANsymphony implements a more sophisticated snapshot mechanism than that of TimeFinder. It also offers Synchronous Data Mirroring for H/A or Business Continuance and Asynchronous Replication for DR.

SANsymphony is a software product that converts an x86 server into a Network Storage Processor. The SANsymphony architecture is N+1, allowing you to go beyond the availability and scalability of the traditional storage array. As a Storage Processor, SANsymphony implements all the advanced features one would expect to find in a storage array. Better still, SANsymphony implements some features you can't find in the traditional array, such as Storage Tiering and QOS (Quality of Service) levels on virtual volume mappings, allowing you to throttle the performance of some of your storage clients, preserving the SAN bandwidth for your critical OLTP databases.

Of course, SANsymphony allows you to implement the SAME (Stripe And Mirror Everything) strategies you've always implemented with the DMX.

Building A Storage Processor

Here's how it works. Take your standard x86 server of choice. Install a clean copy of Windows (XP Pro, 2003 R2, etc.). Install SANsymphony. Reboot. SANsymphony will install its target drivers, I/O subsystem, storage processing and cache management daemons on the server and drive it as a real-time machine. Any available Ethernet ports can be turned into iSCSI targets and/or initiators. Any FC (Fibre Channel) ports can be turned into Fibre Channel targets and/or initiators. The server's memory will be reserved as storage processor cache. Presto.... you have a scalable iSCSI / FC storage array. You are limited only by how much RAM and PCI slots you have on the server. Choose a server with ample PCI slots and you can create a performance monster with more FC ports than your DMX.

A Storage-Agnostic Network Storage Controller

SANsymphony is "Storage Agnostic". Any disks visible in Windows Disk Management on the SANsymphony server can be appropriated by SANsymphony for use in a storage pool. Any disks: EIDE (ATA), SATA, SAS, Ultra320 SCSI, FC, SSA... even USB2 or FireWire (1394) disks are candidates. And of course, any LUNs mapped by your Symmetrix or DMX to the SANsymphony server will appear as disks in Windows Disk Management, ready to be used by SANsymphony. [Once assigned to a SANsymphony storage pool, they become the property of SANsymphony and are no longer visible in Windows Disk Management.]

A Better Capacity Utilization Model

As a Storage Virtualization platform, SANsymphony offers Storage Pooling, Thin Provisioning and Over Subscription. The Thin Provisioning feature will be of particular interest to us in our quest to avoid using BCVs to implement snapshots. Any of the snapshot or replication functions can be implemented by SANsymphony using Thin Provisioning to allocate a virtual volume to serve as the so-called "BCV", a reserved or destination space. A Thin Provisioned volume is a virtual entity, a "skeleton" disk with a geometry, but no physical storage allocated. As data is written to the Thin Provisioned volume, Storage Allocation Units (SAU's) are allocated from the associated storage pool to hold the data. Our Thin Provisioned "BCV" will initially occupy one SAU from the pool.

Implementing Snapshots without TimeFinder

SANsymphony includes a Snapshot Manager that can be used to implement the functions associated with TimeFinder. The Snapshot manager can create multiple snapshot relationships for any or all of SANsymphony's virtual volumes. As SANsymphony is storage agnostic, the source and destination volumes don't even have to be on the same frame. It is perfectly acceptable to create a snapshot using a DMX volume as the source and CX volume as the snapshot destination. Indeed, you could even place the snapshot destination on a JBOD disk, if you were so inclined.

SANsymphony implements a "Copy On Write" Snapshot, meaning that once the snapshot relationship is enabled (e.g. once you take a snapshot), the snapshot volume is immediately available. SANsymphony's Snapshot Manager also allows you to make a complete snapshot clone, as well as roll-forward (Image Update) and roll-back (Source Update) the snapshot. The snapshot destination volumes behave as ordinary volumes: you can mount them, read them, write them. They can be used for implementing backups or for data-mining operations. When they are mapped to a server or servers, their mappings can be assigned QOS (Quality of Service) levels. This means you can "throttle-down" your backup server and take backups of your OLTP volumes during the afternoon without degrading production performance.

Implementing Synchronous Data Mirroring without RDF

SANsymphony implements synchronous data mirroring (RAID-1) between SANsymphony nodes. Like the DMX, SANsymphony can create 2-way and 3-way mirrors. Unlike the DMX, those mirrors can be in physically separate locations, on disparate storage systems. Indeed, a synchronous mirror can be established between any volume on the DMX and any other volume, whether it be on the DMX, an old 8000 series Symm, an FC-4700, a CX3 80, an HDS 99xx, an HP EVA-x000, an IBM DS series, an MD1000, a JBOD. Any volume can be mirrored to any other volume. All volumes appear equal to SANsymphony from lowly SATA drives to high-end DMX LUNs... it's up to you to tier your storage.

SANsymphony is an N+1 architecture, meaning you can have several SANsymphony nodes in a region, all capable of mirroring amongst themselves. Each node is its own active processor "head", contributing cache, ports, CPU, and spindle performance to the matrix in a linear fashion. You can distribute the workload so that in the event of a node failure, you wont have lost half of your storage processing power, as would happen with a traditional dual processor storage array.

Migrating Volumes Between Symm, DMX, Clariion and Other SAN Storage

As stated, SANsymphony is storage agnostic and can be used to synchronously mirror data. Tertiary (or 3-way) mirrors are also possible, as is "replacing" mirror halves. This functionality can be used to migrate data between physical storage arrays, be they Symms, newer DMX's, CX's, AX's, or any other vendor's offerings. As we said, SANsymphony is storage agnostic.

Implementing Asynchronous Replication without RDF

SANsymphony offers an Asynchronous Replication feature known as AIM (Asynchronous IP Mirroring). As the name suggests, AIM works over standard IP and can be used for DR purposes, replicating volumes across long distances over limited bandwidths, similar to the Asynchronous Mode of SRDF. AIM uses standard IP protocols (CIFS/SMB or FTP) over the server's built-in Ethernet ports to implement the transfer of block-level data without requiring expensive protocol converters or using expensive FC or FICON ports.

AIM includes a source and destination buffering mechanism and packet timestamping to assure a coherent replication. The buffering mechanism protects against link failures and allows for planned downtime. AIM includes a mechanism to allow for replication scheduling and bandwidth throttling to make the most out of your VPN.

AIM works with the SANsymphony Snapshot Manager, allowing you to insert snapshot and/or custom script markers in-band into the replication stream to assure coherency points in the replicated data. You can even use the custom markers to kick off backups at the DR site.

AIM is an N-to-N protocol. Multiple AIM relationships can be set up between multiple SANsymphony nodes, and the product is compatible with the SMB SANmelody AIM implementation. [Think of SANmelody as the "CX" to SANsymphony's "DMX"].

Turbo-Charging Your DMX Performance

It seems hard to believe... how can a set of daemons running on a Windows server out-perform a goliath multi-cabinet box with Storage Directors and Channel Directors that cost about $45K a pop? Surely the DMX must be the penultimate in performance, as the price tag would indicate.

Alas, the rules of physics set in. Granted, gravity is only a theory, as is evolution. If you're still fretting over how best to use Hyper-0 on your metas, read on.

Pull out one of those Channel Directors and examine the components. OK, maybe you'd better not. It would test the failover capabilities of the DMX, potentially void your warranty, and maybe cause you headaches with EMC's SLA fine print. But supposing you could pull one of the boards. Any idea what you'd find? Last time I looked at a Symm board there were high-performance 68060's. You remember those Motorola chips? I recall Apple abandoned the Motorola 68K family in the mid 1990s, adopting the more powerful PowerPC.

EMC had the same idea, only several years later. That was the evolution of the Symmetrix. The DMX now sports high-performance PowerPC processors. But Apple is now using Intel. When Apple switched to Intel, they were boasting a 2-3 fold performance increase even in emulation mode. Wonder how much Apple could have boasted if they had gone with AMD Opterons?

SANsymphony runs on the x86 architecture, for better or for worse. As a software engineer, I never did like the little-endian nature of the x86, but heck... Intel won, didn't they? It's a faster architecture and it keeps getting faster.

But processor speeds aren't the most important element of a SAN storage processor. Memory access speeds and I/O bus speeds and peripheral bus speeds are far more important. Look on that DMX Channel Director. Are there PCI Express connections on it? No? How fast is the memory? Oh... that's on another card, across another bus? Is it 800 MHz memory? 667 MHz? Not even?

Many of the rack-mount servers shipping today have at least 2 x 8-lane PCI Express slots... ideal for installing the latest 4-port PCI Express cards from QLogic capable of sustaining 150K IOPS per port.

Add it all up... the DMX is not as fast as the server you just bought for testing VMWare ESX 3.0.1...

By the way, does your DMX support 2Gb or 4Gb FC? SANsymphony can talk 2Gb on the back end to your Channel Directors and 4Gb on the front end to your OLTP servers.

Seriously, a POC will prove definitively that a SANsymphony implementation on recent x86 platform will run circles around the DMX it's fronting. Chances are you will not want to uninstall the POC when you see the spectacular performance gains. [For you techies, here's a bit of trivia.... 20 usec round trip on a cache hit. 40 on a mirrored-write cache hit. Ask EMC for some performance numbers on the DMX.... they ain't talkin'...]

Proxy Volumes For Minimum Disruption of Your Existing SAN

SANsymphony has a unique optional feature called Proxy Volumes. With Proxy Volumes, you can map an existing LUN through SANsymphony by Proxy. SANsymphony won't disturb the geometry of the existing LUN, but will merely behave as a pass-through. The Proxy Volume gets many of the benefits of a standard virtualized SANsymphony volume: Caching, Snapshots, Data Migration and Synchronous Data Mirroring to some other storage, Asynchronous Replication for DR, QOS management. Of course, a Proxy Volume can't take advantage of SANsymphony's Thin Provisioning or Storage Pooling. Nonetheless, it is a great way to implement data migration of LUNs with minimal downtime.

Limitations — What SANsymphony Can't Do

SANsymphony is an open storage platform limited to presenting SCSI protocol devices, such as Fibre Channel and iSCSI. While SANsymphony can coexist in a mixed DMX world (AS-400's and Open Systems), it is unlikely that SANsymphony will be able to serve volumes to your AS-400 or mainframe machines. While it may be possible that SANsymphony can be used to replicate BCVs of AS-400 volumes to a remote site, SANsymphony does not speak the disk I/O language of the AS-400. [Nor do I, for that matter!] However, SANsymphony is compatible with all "Open Systems" OS's, including Windows, Linux, VMWare ESX, Solaris, Netware, AIX, HP/UX, DG/UX (anyone remember that?), and even Mac OS (both 9.x and X.x).

En resume... en conclusion...

DataCore Software Corporation has been in business since 1998 as a privately held (employee-owned) company. The SANsymphony product has been shipping since the year 2000 and is a mature, rock-solid Network Storage Controller already deployed in enterprises all over the world. The company has a significant presence in Healthcare, Finance, Manufacturing, Retailing and eCommerce. The SANsymphony product is ideally suited to environments incorporating Tier-1 systems such as the EMC Symmetrix and DMX. What's more, a complete SANsymphony configuration suitable for a DMX data-center costs a small fraction of what you would pay for the TimeFinder and RDF features. [In fact, a three node configuration complete with 3 years maintenance and several days of PS will most likely cost less than the annual support you're paying on TimeFinder and RDF alone!]

Contact Las Solanas Consulting or DataCore Software Corporation today for more information. The DataCore sales staff are friendly, non-pushy, technically competent people who can answer any of your questions and, if you so desire, arrange a POC or Try and Buy.