Will Storage Virtualization Degrade I/O Performance on my SAN?
That depends entirely on the virtualization implementation. Much has been written about the two common
methods of implementing storage virtualization, "In-Band" and "Out-Of-Band". The arguments are
generally limited to the concepts and not the actual implementations by the vendors. Caveat emptor.
In-Band and Out-Of-Band refer to how storage virtualization impacts the actual I/O traffic between
the host server's OS (file system) and the presumed storage array that houses the physical disks. In-Band
virtualization implies that the virtualization engine is in the data path, receives the I/O requests
from the OS, and completes those requests to the storage array on behalf of the host. Out-Of-Band
virtualization supposedly means that the virtualization layer somehow does not impact I/O traffic; that
redirection of I/O's from file system requests to physical storage is happening somewhere outside the
data path, either on the host itself (an agent) or on some external device that interfaces with the
SAN switches or storage array.
The argument that Out-Of-Band does in no way adversly affect performance is bogus. Any agent
running on a host directing traffic is going to impact the performance of that host and potentially
impact stability of the system. If the virtualization is implemented in the switch, then the traffic
redirection will potentially add microseconds of latency to the switching. Anyone with experience
in network switches, be they Ethernet or Fibre Channel, will tell you that switches add latency
some more than others depending on their implementation. If the virtualization
is occuring within the storage array itself, it can be definitively argued that it does not affect
performance, but it can equally be argued that storage array-based virtualization is in fact In-Band
(the storage processor is in the data path, sitting between the host file system and the physical disks)
and its value (i.e. feature set) is limited to the array itself.
The arguments against In-Band typically center around performance, with the idea that an
In-Band virtualization engine will add an "extra hop" and become the bottleneck of the SAN.
These arguments are too generalized and simplistic; the performance of In-Band virtualization
must be analyzed on an implementation basis, case by case.
If the In-Band implementation is nothing more than a traffic redirection, then clearly,
it will add latency to all I/Os, just as would the so-called "Out-Of-Band" virtualization
implemented in a switch.
However, if the In-Band implementation includes an adaptive caching engine, then ostensibly
it is adding a new layer of cache between the file system and the storage array. Effectively,
an In-Band Storage Virtualization engine with caching is a first-line Storage Processor or
a "Network Storage Processor".
Increasing the quantity and fan out of cache layers will almost invariably improve performance.
This is the reason CPUs typically have 3 levels of cache, servers perform better with more RAM
(which is also a form of cache), and storage vendors advise increasing the amount of cache on
the array's storage processors to improve performance. The goal is always to avoid going
to the slowest device in the I/O path: the spindles.
Consider that most physical disks typically require at least 3000 µSec to complete an I/O request,
due to rotational latency and head seek times. The storage processor in a storage array uses
caching to mask some of the latency; this can pay off handsomely depending on the nature of the I/Os.
Most high-end storage processors complete I/O requests in 200-400 µSec on a cache hit. Contrast that
with the typical latency associated with pulling data from a server's local memory: sub 2 µSec.
The I/O latency associated with a cache hit on a high-end storage processor (or any storage processor,
for that matter) can be attributed to several factors:
- bandwidth of front end targets (iSCSI? 1, 2, or 4Gb Fibre Channel?)
- internal I/O bus speeds
- CPU or processor performance
- memory and memory bus speeds
- level of optimization of the storage processor code
- foundation OS the storage processor code runs on
- RAID engine (software or hardware?)
- I/O model used (interrupt driven or polling?)
Most of today's low-end and mid-range storage arrays (or SANs as they are often misnamed)
are based on commodity components, built on a Wintel platform: an Intel-compatible motherboard
running some form of Windows, typically "Windows Embedded". This allows the storage manufacturer
to more rapidly turn out a new array with new features without having to port code, write new
drivers, etc. But keep in mind the laws of "Time To Market". In general, it takes 12-18 months
from product conception to product ship. Product conception begins with determining price points
and market demand for a product that will ship in 12-18 months, determining its feature set, and
then costing out the necessary parts and quantities. Early on, the negotiation with suppliers
locks down costs and quantities... on specific part numbers (a.k.a. the "BOM" or Bill of Materials).
By the time the storage array ships, it is based on parts and technologies that are 12-18 months old.
It's not surprising that 4Gb Fibre Channel HBA's showed up on the market 12 months before the first
4Gb Fibre Channel storage arrays shipped.
Applying this understanding to the question of performance in In-Band storage virtualization
we can begin to perceive that a cache-based In-Band engine based on recent technologies should
be able to perform faster than a storage array based on 12-18 month old technologies.
Example: Build a cache-based In-Band Storage Virtualization engine using a late-model x86 server
with adequate RAM (2 to 4GB). Add in recent technology (e.g. 4Gb) Fibre Channel HBA's to be used
as "targets" or Front-End Ports. Upgrade your application server HBA's, etc., to 4Gb technology.
Install a well-designed, caching Storage Virtualization software package on the server. Place this
Storage Virtualization server between your application servers and your existing expensive,
optimized, not-yet-depreciated, 2Gb FC storage array. Choose a conservative I/O mix that shouldn't
raise the eyebrows of most storage admins: 70% reads to 30% writes. Given similar channel fan-out
to your application servers, you can be guaranteed that performance will nearly double.
Don't take our word for it try it for yourself.
DataCore Software Corporation offers
no-obligation 30-day evaluation of their SANmelody™
Storage Server software. SANmelody™
turns an ordinary Windows-based server into a Virtual SAN
or Storage Array. This evaluation software supports both iSCSI and Fibre Channel: iSCSI comes for free, using the server's built-in ethernet
ports; for Fibre Channel support you will need to add one or more Fibre Channel HBAs to the server.
Add sufficient RAM to implement caching with the product as you would with a traditional storage
array. If you need a benchmark application, try the IOmeter
utility available as a free download from the web.
The performance results you will achieve with SANmelody will, of course, depend on factors such
as those listed in the bullet points above this isn't rocket science. The faster your PCI bus
speeds, the choice of Fibre Channel HBA technology, the amount of RAM (cache) in the server will
all affect performance. An "apples to apples" evaluation will give you an opportunity to draw your
own conclusions about what factors most impact performance in a SAN and give you a better grasp on
the "In-Band" vs. "Out-Of-Band" arguments.