Effective Data Deduplication Implementation

Effective Data Deduplication Implementation

Enterprises with IT infrastructure are looking at reducing their carbon foot print and infrastructure management cost by slimming down their data centers. In contrast, data volumes are growing exponentially in the enterprise world, leading to a surging demand for capacity and new hardware. Data Deduplication is one of the techniques that can help enterprises to have lesser store space to store the same amount of data. Simply put Deduplication means removal of duplicates from data. Deduplication can provide 2 to 200 times space savings and can also be used for low bandwidth data transfer. However it is a resource intensive operation and can offset the advantages if not implemented correctly.

Deduplication is a resource intensive operation and different vendors have different ways of implementation. The challenge is to achieve maximum dedupe ratio (also called as data compression ratio) with as little affect on throughput (IOPS) as possible

Irrespective of vendor implementation, Data Deduplication can be divided into four steps of data segmentation, creating finger prints for these data segments, matching these finger prints for duplicates and storing unique segments on disk. Each of the above mentioned steps can be implemented using different techniques and each one of them has their own advantages and disadvantages. This white paper will discuss various techniques of implementing the above mentioned steps and their effect on Data Deduplication ratio and throughput.

Data Deduplication Defined

Data Deduplication is the new data compaction technology which removes duplicates in data. SNIA defines it as “Data Deduplication is the process of examining a data-set or I/O stream at the sub-file level and storing and /or sending only unique data”. It differs from the compression techniques by working on the data at sub-file level where as compression encodes the data in the file to reduce its storage requirement. However, compression can be used to augment data deduplication to provide higher dedupe ratio (Size of data to be written / Size of data on disk: 1) Small to large enterprises have been adopting this new technology as it gives significant Return on Investment by:

• Reducing the storage capacity required to store the data.
• Reducing network bandwidth required to transfer the data.

The storage device cost has been reducing with the advance in disk technology. On the other end, the IO bandwidth is on the upswing with Fibre Channel (FC) based networks, Gigabit Ethernet and iSCSI. IO throughput of these storage devices has increased and cost per GB of storing data has reduced. Data Deduplication does reduce the cost per GB of data even further but can be taxing on the storage device resources due to its mathematically extensive operations. Efficiency of any Data Deduplication application is measured by the Dedupe ratio (Size of Actual Data / Size of Data after deduplication: 1) and throughput (Megabytes of Data Deduplicated per sec). Following are the parameters affecting dedupe ratio and throughput:

• Nature of data to be deduplicated.
• Where is Deduplication applied? Data Deduplication can be applied either on source appliance or on target appliance.
• If Data Deduplication is inline or a post processing application?
• Implementation of Data Deduplication.

The following sections will detail out generic steps in Data Deduplication and how their implementation has a bearing on the Dedupe ratio and Deduplication throughput.

Download

Download full paper At
http://www.enjineer.com/forum