How to Implement Snapshots, Data Mirroring and Replication on Symmetrix and DMX storage without using BCVs, TimeFinder or RDF
A White Paper by Tim Warden
November 17, 2006
Despite their unconventional start as an office furniture supplier, EMC made its name with their Symmetrix (a.k.a. DMX) product line.
These Tier-1 SAN storage servers offer a rich feature set including snapshots and data replication, not to mention performance
tuning features. Unfortunately, those advanced features come at a significant price, both in licensing costs and capacity utilization.
EMC licenses each feature across the entire storage park. If you have 4 loaded DMX servers in your data-center
and you want to implement TimeFinder for a few of the volumes on one of the arrays, you have to buy the license for the whole
installation... plus annual support. It's not uncommon to hear of customers spending over $2 million to implement TimeFinder
and RDF in a large data-center. A very costly undertaking.
But the costs are not limited to licensing and support charges.
DMX storage admins are only too familiar with the dreaded BCV or Business Continuance Volume, a ravenous monster that
consumes mass quantities of your expensive Tier-1 storage in order to implement the advanced features of TimeFinder and
RDF. Like ordinary volumes, BCV volumes are based on the meta devices of the DMX. If you wish to make a TimeFinder copy
of a particular volume, you must create a BCV volume of equal size. If you wish to make TimeFinder copies of all your
volumes, you'll need to allocate half your logical disk space to accommodate all those BCVs. If your meta volumes are
based on RAID-S group hypers, you pay an additional penalty in parity disk space. In the extreme case that all your meta
volumes are based on RAID-1 mirrored hypers... only 1/4 of your Tier-1 storage is effectively usable space.
RDF or SRDF further complicates matters. The data replication features requires a compatible Symmetrix or DMX, and of
course BCVs are used on both source and destination Symms to implement the replication.
Storage Virtualization To The Rescue
As it turns out, two of the advanced features of the Symmetrix / DMX can be found in Storage Virtualization products
such as SANsymphony from
DataCore Software Corporation.
SANsymphony implements a more
sophisticated snapshot mechanism than that of TimeFinder. It also offers Synchronous Data Mirroring for H/A or Business
Continuance and Asynchronous Replication for DR.
SANsymphony is a software product
that converts an x86 server into a Network Storage Processor. The SANsymphony architecture is N+1, allowing you to go
beyond the availability and scalability of the traditional storage array. As a Storage Processor, SANsymphony implements
all the advanced features one would expect to find in a storage array. Better still, SANsymphony implements some features
you can't find in the traditional array, such as Storage Tiering and QOS (Quality of Service) levels on virtual volume
mappings, allowing you to throttle the performance of some of your storage clients, preserving the SAN bandwidth for
your critical OLTP databases.
Of course, SANsymphony allows you to implement the SAME (Stripe And Mirror Everything) strategies you've always
implemented with the DMX.
Building A Storage Processor
Here's how it works. Take your standard x86 server of choice. Install a clean copy of Windows (XP Pro, 2003 R2, etc.).
Install SANsymphony. Reboot. SANsymphony will install its target drivers, I/O subsystem, storage processing and cache management
daemons on the server and drive it as a real-time machine. Any available Ethernet ports can be turned into iSCSI targets and/or
initiators. Any FC (Fibre Channel) ports can be turned into Fibre Channel targets and/or initiators. The server's memory
will be reserved as storage processor cache. Presto.... you have a scalable iSCSI / FC storage array. You are limited only
by how much RAM and PCI slots you have on the server. Choose a server with ample PCI slots and you can create a performance
monster with more FC ports than your DMX.
A Storage-Agnostic Network Storage Controller
SANsymphony is "Storage Agnostic". Any disks visible in Windows Disk Management on the SANsymphony server can be appropriated
by SANsymphony for use in a storage pool. Any disks: EIDE (ATA), SATA, SAS, Ultra320 SCSI, FC, SSA... even USB2 or FireWire (1394)
disks are candidates. And of course, any LUNs mapped by your Symmetrix or DMX to the SANsymphony server will appear as disks
in Windows Disk Management, ready to be used by SANsymphony. [Once assigned to a SANsymphony storage pool, they become the
property of SANsymphony and are no longer visible in Windows Disk Management.]
A Better Capacity Utilization Model
As a Storage Virtualization platform, SANsymphony offers
Storage Pooling, Thin Provisioning and
Over Subscription. The Thin Provisioning feature will be of particular interest to us in our quest to avoid using
BCVs to implement snapshots. Any of the snapshot or replication functions can be implemented by SANsymphony using
Thin Provisioning to allocate a virtual volume to serve as the so-called "BCV", a reserved or destination space.
A Thin Provisioned volume is a virtual entity, a "skeleton" disk with a geometry, but no physical storage allocated.
As data is written to the Thin Provisioned volume, Storage Allocation Units (SAU's) are allocated from the associated
storage pool to hold the data. Our Thin Provisioned "BCV" will initially occupy one SAU from the pool.
Implementing Snapshots without TimeFinder
SANsymphony includes a Snapshot Manager that can be used to implement the functions associated with TimeFinder. The Snapshot
manager can create multiple snapshot relationships for any or all of SANsymphony's virtual volumes. As SANsymphony is storage
agnostic, the source and destination volumes don't even have to be on the same frame. It is perfectly acceptable to create
a snapshot using a DMX volume as the source and CX volume as the snapshot destination. Indeed, you could even place the
snapshot destination on a JBOD disk, if you were so inclined.
SANsymphony implements a "Copy On Write" Snapshot, meaning that once the snapshot relationship is enabled (e.g. once you
take a snapshot), the snapshot volume is immediately available. SANsymphony's Snapshot Manager also allows you to make a complete
snapshot clone, as well as roll-forward (Image Update) and roll-back (Source Update) the snapshot. The snapshot destination
volumes behave as ordinary volumes: you can mount them, read them, write them. They can be used for implementing backups
or for data-mining operations. When they are mapped to a server or servers, their mappings can be assigned QOS (Quality of Service)
levels. This means you can "throttle-down" your backup server and take backups of your OLTP volumes during the afternoon without
degrading production performance.
Implementing Synchronous Data Mirroring without RDF
SANsymphony implements synchronous data mirroring (RAID-1) between SANsymphony nodes. Like the DMX, SANsymphony can create 2-way
and 3-way mirrors. Unlike the DMX, those mirrors can be in physically separate locations, on disparate storage systems. Indeed,
a synchronous mirror can be established between any volume on the DMX and any other volume, whether it be on the DMX, an old 8000
series Symm, an FC-4700, a CX3 80, an HDS 99xx, an HP EVA-x000, an IBM DS series, an MD1000, a JBOD. Any volume can be mirrored to any other
volume. All volumes appear equal to SANsymphony from lowly SATA drives to high-end DMX LUNs... it's up to you to tier your storage.
SANsymphony is an N+1 architecture, meaning you can have several SANsymphony nodes in a region, all capable of mirroring amongst
themselves. Each node is its own active processor "head", contributing cache, ports, CPU, and spindle performance to the matrix in
a linear fashion. You can distribute the workload so that in the event of a node failure, you wont have lost half of your storage
processing power, as would happen with a traditional dual processor storage array.
Migrating Volumes Between Symm, DMX, Clariion and Other SAN Storage
As stated, SANsymphony is storage agnostic and can be used to synchronously mirror data. Tertiary (or 3-way) mirrors are also
possible, as is "replacing" mirror halves. This functionality can be used to migrate data between physical storage arrays, be
they Symms, newer DMX's, CX's, AX's, or any other vendor's offerings. As we said, SANsymphony is storage agnostic.
Implementing Asynchronous Replication without RDF
SANsymphony offers an Asynchronous Replication feature known as AIM (Asynchronous IP Mirroring). As the name suggests, AIM
works over standard IP and can be used for DR purposes, replicating volumes across long distances over limited bandwidths,
similar to the Asynchronous Mode of SRDF. AIM uses standard IP protocols (CIFS/SMB or FTP) over the server's built-in Ethernet
ports to implement the transfer of block-level data without requiring expensive protocol converters or using expensive FC or
FICON ports.
AIM includes a source and destination buffering mechanism and packet timestamping to assure a coherent replication. The buffering
mechanism protects against link failures and allows for planned downtime. AIM includes a mechanism to allow for replication scheduling
and bandwidth throttling to make the most out of your VPN.
AIM works with the SANsymphony Snapshot Manager, allowing you to insert snapshot and/or custom script markers in-band into the
replication stream to assure coherency points in the replicated data. You can even use the custom markers to kick off backups at
the DR site.
AIM is an N-to-N protocol. Multiple AIM relationships can be set up between multiple SANsymphony nodes, and the product is
compatible with the SMB SANmelody AIM implementation. [Think of SANmelody as the "CX" to SANsymphony's "DMX"].
Turbo-Charging Your DMX Performance
It seems hard to believe... how can a set of daemons running on a Windows server out-perform a goliath multi-cabinet
box with Storage Directors and Channel Directors that cost about $45K a pop? Surely the DMX must be the penultimate in
performance, as the price tag would indicate.
Alas, the rules of physics set in. Granted, gravity is only a theory, as is evolution. If you're still fretting over how best
to use Hyper-0 on your metas, read on.
Pull out one of those Channel Directors and examine the components. OK, maybe you'd better not. It would test the failover
capabilities of the DMX, potentially void your warranty, and maybe cause you headaches with EMC's SLA fine print. But supposing
you could pull one of the boards. Any idea what you'd find? Last time I looked at a Symm board there were high-performance 68060's.
You remember those Motorola chips? I recall Apple abandoned the Motorola 68K family in the mid 1990s, adopting the more powerful
PowerPC.
EMC had the same idea, only several years later. That was the evolution of the Symmetrix. The DMX now sports high-performance
PowerPC processors. But Apple is now using Intel. When Apple switched to Intel, they were boasting a 2-3 fold performance increase
even in emulation mode. Wonder how much Apple could have boasted if they had gone with AMD Opterons?
SANsymphony runs on the x86 architecture, for better or for worse. As a software engineer, I never did like the little-endian
nature of the x86, but heck... Intel won, didn't they? It's a faster architecture and it keeps getting faster.
But processor speeds aren't the most important element of a SAN storage processor. Memory access speeds and I/O bus speeds
and peripheral bus speeds are far more important. Look on that DMX Channel Director. Are there PCI Express connections on it?
No? How fast is the memory? Oh... that's on another card, across another bus? Is it 800 MHz memory? 667 MHz? Not even?
Many of the rack-mount servers shipping today have at least 2 x 8-lane PCI Express slots... ideal for installing the latest
4-port PCI Express cards from QLogic capable of sustaining 150K IOPS per port.
Add it all up... the DMX is not as fast as the server you just bought for testing VMWare ESX 3.0.1...
By the way, does your DMX support 2Gb or 4Gb FC? SANsymphony can talk 2Gb on the back end to your Channel Directors and 4Gb
on the front end to your OLTP servers.
Seriously, a POC will prove definitively that a SANsymphony implementation on recent x86 platform will run circles around the
DMX it's fronting. Chances are you will not want to uninstall the POC when you see the spectacular performance gains. [For you
techies, here's a bit of trivia.... 20 usec round trip on a cache hit. 40 on a mirrored-write cache hit. Ask EMC for some
performance numbers on the DMX.... they ain't talkin'...]
Proxy Volumes For Minimum Disruption of Your Existing SAN
SANsymphony has a unique optional feature called Proxy Volumes. With Proxy Volumes, you can map an existing LUN through SANsymphony
by Proxy. SANsymphony won't disturb the geometry of the existing LUN, but will merely behave as a pass-through. The Proxy Volume
gets many of the benefits of a standard virtualized SANsymphony volume: Caching, Snapshots, Data Migration and Synchronous Data
Mirroring to some other storage, Asynchronous Replication for DR, QOS management. Of course, a Proxy Volume can't take advantage of
SANsymphony's Thin Provisioning or Storage Pooling. Nonetheless, it is a great way to implement data migration of LUNs with
minimal downtime.
Limitations — What SANsymphony Can't Do
SANsymphony is an open storage platform limited to presenting SCSI protocol devices, such as Fibre Channel and iSCSI. While SANsymphony
can coexist in a mixed DMX world (AS-400's and Open Systems), it is unlikely that SANsymphony will be able to serve volumes to
your AS-400 or mainframe machines. While it may be possible that SANsymphony can be used to replicate BCVs of AS-400 volumes
to a remote site, SANsymphony does not speak the disk I/O language of the AS-400. [Nor do I, for that matter!] However, SANsymphony
is compatible with all "Open Systems" OS's, including Windows, Linux, VMWare ESX, Solaris, Netware, AIX, HP/UX, DG/UX (anyone remember
that?), and even Mac OS (both 9.x and X.x).
En resume... en conclusion...
DataCore Software Corporation has been in business since 1998 as a privately held (employee-owned) company. The SANsymphony
product has been shipping since the year 2000 and is a mature, rock-solid Network Storage Controller already deployed in enterprises
all over the world. The company has a significant presence in Healthcare, Finance, Manufacturing, Retailing and eCommerce. The
SANsymphony product is ideally suited to environments incorporating Tier-1 systems such as the EMC Symmetrix and DMX. What's
more, a complete SANsymphony configuration suitable for a DMX data-center costs a small fraction of what you would pay for the
TimeFinder and RDF features. [In fact, a three node configuration complete with 3 years maintenance and several days of PS will
most likely cost less than the annual support you're paying on TimeFinder and RDF alone!]
Contact Las Solanas Consulting or
DataCore Software Corporation today for more information.
The DataCore sales staff are friendly, non-pushy, technically competent people who can answer any of your questions
and, if you so desire, arrange a POC or Try and Buy.
|
