VMware Storage I/O Control

1. Overview

Storage I/O Control (SIOC) was initially introduced in vSphere 4.1 to provide I/O prioritization of virtual machines running on a cluster of ESXi hosts that had access to  shared storage.  It extended the familiar constructs of shares and limits, which existed for CPU and memory, to address storage utilization through a dynamic allocation of I/O queue slots across a cluster of ESXi servers. The purpose of SIOC is to address the ‘noisy neighbour’ problem, i.e. a low priority virtual machine impacting other higher priority virtual machines due to the nature of the application and its I/O running in that low priority VM.

vSphere 5.0 extended Storage I/O Control (SIOC) to provide cluster-wide I/O shares and limits for NFS datastores.

vSphere 5.1 introduced Automatic Threshold Computation enhancement to SIOC.

To figure out the best latency threshold (default 30 msecs), the new automatic threshold detection uses the I/O injector modelling functionality of SIOC to determine what the peak throughput of a datastore is. When peak throughput is measured, latency is also measured.

The latency threshold value at which Storage I/O Control will kick in is then set to 90% of this peak value (by default). vSphere administrators can change this 90% to another percentage value or they can still input a millisecond value if they so wish.

See also:  Storage I/O fairness and vSphere Resource Management Guide

2. Configuration

2.1   Enable SIOC and set the congestion threshold for datastore

2.2 Configure disk shares for VM

3. How it works

Ref:

VMware Storage I/O Control whitepaper

VMworld Storage I/O Control session

SIOC monitors the latency of I/Os to datastores at each ESX host sharing that device. When the average normalized datastore latency exceeds a set threshold (30ms by default), the datastore is considered to be congested, and SIOC kicks in to distribute the available storage resources to virtual machines in proportion to their shares. This is to ensure that low-priority workloads do not monopolize or reduce I/O bandwidth for high-priority workloads. SIOC accomplishes this by throttling back the storage access of the low-priority virtual machines by reducing the number of I/O queue slots available to them. Depending on the mix of virtual machines running on each ESX server and the relative I/O shares they have, SIOC may need to reduce the number of device queue slots that are available on a given ESX server.

See also: how SIOC calculates latency across all hosts.

4. Best practices

4.1 Adaptive Queueing vs. Storage I/O Control

With adaptive queueing (introduced in VMware ESX 3.5 Update 4), on receipt of the queue full status, the LUN queue depth of the host to that device is halved.

See also: Controlling LUN queue depth throttling in VMware ESX/ESXi (VMware KB 1008113).

4.2 SIOC and datastores spread across all spindles in the array

See also: Unmanaged I/O workload detected on shared datastore running Storage I/O Control (SIOC) for congestion management (VMware KB 1020651).

4.3 Debunking Storage I/O Control Myths

VMware VM Storage vMotion

1. Overview

VMware® Storage VMotion™ enables live migration for running virtual machine disk files from one storage location to another with no downtime or service disruption:

• Simplify storage array migrations and storage upgrades.
• Dynamically optimize storage I/O performance.
• Efficiently utilize storage and manage capacity.

VMware vSphere® 5.1 vMotion presents a unified and integrated migration architecture that migrates live virtual machines, including their memory and storage, between vSphere hosts without any requirement for shared storage. When a virtual machine needs to be migrated between hosts that do not share storage, vMotion copies the data across the vMotion network using some of the Storage vMotion code. This shared-nothing live migration feature offers administrators significantly more simplicity and flexibility in managing and moving virtual machines across their virtual infrastructures compared to the traditional vMotion and Storage vMotion migration solutions.

Prior to 5.1, Storage vMotion solution was limited to the hosts that shared a common set of datastores. In addition, migration of an entire virtual machine required two separate operations, for instance vMotion followed by Storage vMotion, or vice versa.

2. How it works

Ref:

VMware vSphere® 5.1 vMotion Architecture, Performance and Best Practices

Insight Into vMotion: Architectures, Performance, Best Practices, and Futures

The Design and Evolution of Live Storage Migration in VMware ESX

2.1 Storage vMotion (vSphere 5.1, 2012) – asynchronous transport mechanism

2.1.1 non-shared datastore migration – asynchronous transport mechanism

vSphere 5.1 Storage vMotion follows an almost identical model for migrating a virtual disk, but uses a network transport for migrating the data. In contrast to previous Storage vMotion, vSphere 5.1 Storage vMotion cannot rely on synchronous storage mirroring because the source and destination datastores might be separated by longer physical distances. High or varying network latency can impact the virtual machine I/O performance adversely when using synchronous storage replication. Instead, vSphere 5.1 Storage vMotion relies on an asynchronous transport mechanism for migrating both the bulk copy process and I/O mirroring process data. The asynchronous transport uses a buffering technique that essentially enables vSphere 5.1 Storage vMotion to maintain very good virtual machine performance during the storage migration even in the face of higher and unpredictable network latencies.

vSphere 5.1 Storage vMotion typically transfers the disk content over the vMotion network. However vSphere 5.1 Storage vMotion optimizes the disk copy by leveraging the mechanisms of Storage vMotion whenever possible. For instance, if the source host has access to the destination datastore, vSphere 5.1 Storage vMotion will use the source host’s storage interface to transfer the disk content, thus reducing vMotion network utilization and host CPU utilization. Similarly, if both the source and destination datastores are on the same array that is capable of using VMware vSphere® Storage APIs – Array Integration (VAAI), and the source host has access to the destination datastore, vSphere 5.1 Storage vMotion will offload the task of copying the disk content to the array using VAAI.

2.1.2 Parallel Storage vMotions

vSphere 5.1 allows up to 4 parallel disk copies per Storage vMotion operation, where previous versions of vSphere used to copy disks serially. When you migrate a virtual machine with five VMDK files, Storage vMotion copies of the first four disk in parallel, then start the next disk copy as soon as one of the first four finish.

To reduce performance impact on other virtual machines sharing the datastores, parallel disk copies only apply to disk copies between distinct datastores. This means that if a virtual machine has multiple VMDK files on datastore A and B, parallel disk copies will only happen if destination datastores are C and D.

A fan out disk copy, in other words copying two VMDK files on datastore A to datastores B and C, will not have parallel disk copies. The common use case of parallel disk copies is the migration of a virtual machine configured with an anti-affinity rule inside a datastore cluster.

2.2 Storage vMotion (vSphere 5.0, 2011) – Synchronous IO Mirroring method

2.2.1 A new Storage vMotion mechanism based on a mirror driver

Storage vMotion uses a synchronous mirroring approach to migrate a virtual disk from one datastore to another datastore on the same physical host. This is implemented by using two concurrent processes. First, a bulk copy (also known as a clone) process (data mover) proceeds linearly across the virtual disk in a single pass and performs a bulk copy of the disk contents from the source datastore to the destination datastore. Concurrently, an I/O mirroring process transports any additional changes that occur to the virtual disk, because of the guest’s ongoing modifications. The I/O mirroring process accomplishes that by mirroring the ongoing modifications to the virtual disk on both the source and the destination datastores.  When the guest OS of the virtual machine that is undergoing the process using Storage vMotion initiates a write to an already copied block, the mirror driver will synchronously mirror this write and wait for both acknowledgements before communicating this to the guest OS.

Storage vMotion mirrors I/O only to the disk region that has already been copied by the bulk copy process. Guest writes to a disk region that the bulk copy process has not yet copied are not mirrored because changes to this disk region will be copied by the bulk copy process eventually.

A synchronization mechanism is implemented that prevents the guest write I/Os from conflicting with the bulk copy process read I/Os when the guest write I/Os are issued to the disk region currently being copied by the bulk copy process. No synchronization is needed for guest read I/Os, which are issued only to the source virtual disk.

The Storage vMotion process is fairly straight forward:

1)     The virtual machine working directory is copied by VPXA to the destination datastore.

2)     A “shadow” virtual machine is started on the destination datastore using the copied files. The “shadow” virtual machine idles, waiting for the copying of the virtual machine disk file(s) to complete.

3)     Storage vMotion enables the Storage vMotion Mirror driver to mirror writes of already copied blocks to the destination.

4)     In a single pass, a copy of the virtual machine disk file(s) is completed to the target datastore while mirroring I/O.

5)     Storage vMotion invokes a Fast Suspend and Resume of the virtual machine (similar to vMotion) to transfer the running virtual machine over to the idling shadow virtual machine.

6)     After the Fast Suspend and Resume completes, the old home directory and VM disk files are deleted from the source datastore.

It should be noted that the shadow virtual machine is only created in the case that the virtual machine home directory is moved. If and when it is a “disks-only Storage vMotion, the virtual machine will simply be stunned and unstunned.

2.2.2 Storage vMotion snapshot support.

Storage vMotion in vSphere 5.0 now also supports the migration of virtual machines with a vSphere snapshot and the migration of linked clones.

2.3 Storage vMotion (vSphere 4.1, 2010) – hardware acceleration with VAAI

With Hardware Acceleration with vStorage APIs for Array Integration (VAAI), ESX can offload specific storage operations to compliant storage hardware. With storage hardware assistance, ESX performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.

2.4 Storage vMotion (vSphere 4.0, 2009) – Dirty Block tracking (DBT) method

vSphere 4.0 introduces several new capabilities to Storage VMotion. When Storage vMotion was introduced in VI3 release 3.5, it had a few limitations which vSphere 4.0 addresses.

2.4.1 Fully integration with vCenter

2.4.2 Fully support for migration across datastores of several protocol choices (FC, iSCSI, or NFS)

2.4.3 Fully support of virtual disk format choices (thin or thick)

2.4.4 More efficient migration process – Dirty Block tracking method

The enhancements to Storage VMotion also include a more efficient migration process, as well as the removal of the 3.5 requirement for having 2 times the CPU and memory needed for the VM that is being migrated. Storage VMotion in vSphere now leverages a change block tracking method that moves the VM home over to the new datastore in a few iterations instead of using the virtual disk snapshot method used in the 3.5 release.

Before moving a virtual machines disk file, Storage VMotion moves the “home directory” of the virtual machine to the new location. The home directory contains meta data about the virtual machine (configuration, swap and log files). After relocating the home directory, Storage VMotion copies the contents of the entire virtual machine storage disk file to the destination storage host, leveraging “changed block tracking” to maintain data integrity during the migration process. Next, the software queries the changed block tracking module to determine what regions of the disk were written to during the first iteration, and then performs a second iteration of copy, where those regions that were changed during the first iteration copy (there can be several more iterations).

Once the process is complete, the virtual machine is quickly suspended and resumed so that it can begin using the virtual machine home directory and disk file on the destination datastore location. Before VMware ESX allows the virtual machine to start running again, the final changed regions of the source disk are copied over to the destination and the source home and disks are removed.

This approach guarantees complete transactional integrity and is fast enough to be unnoticeable to the end user.

2.5 Storage vMotion (vSphere 3.5, 2007) – Snapshot based method

http://technodrone.blogspot.com/2011/05/storage-vmotion-deep-dive.html

http://www.vmworld.com/docs/DOC-2931

 

3. Storage vMotion with RDM

Ref:

VMware KB 1005241

http://blogs.vmware.com/vsphere/2012/02/migrating-rdms-and-a-question-for-rdm-users.html

3.1 VM with Physical (Pass-Thru) RDMs (Powered On – Storage vMotion):

• If I try to change the format to thin or thick, then no Storage vMotion allowed.
• If I chose not to do any conversion, only the pRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN.

3.2 VM with Virtual (non Pass-Thru) RDMs (Power On – Storage vMotion):

• On a migrate, if I chose to covert the format in the advanced view, the vRDM is converted to a VMDK on the destination VMFS datastore.
• If I chose not to do any conversion, only the vRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN (same behaviour as pRDM)

3.3 VM with Physical (Pass-Thru) RDMs (Powered Off – Cold Migration):

• On a migrate, if I chose to change the format (via the advanced view), the pRDM is converted to a VMDK on the destination VMFS datastore.
• If I chose not to do any conversion, only the pRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN

3.4 VM with Virtual (non Pass-Thru) RDMs (Power Off – Cold Migration):

• On a migrate, if I chose to covert the format in the advanced view, the vRDM is converted to a VMDK on the destination VMFS datastore.
• If I chose not to do any conversion, only the vRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN (same behaviour as pRDM).

As you can see, there are 3 occasions when an RDM could be converted to a VMDK. Perhaps the most surprising is the fact that a pRDM could be converted to a VMDK, when a cold migration of the VM is attempted, and the format is changed.

VMware VM Live Migration (vMotion)

1. Overview

vSphere’s live migration (vMotion) allows you to move an entire running virtual machine from one physical server to another, without downtime. The virtual machine retains its network identity and connections, ensuring a seamless migration process. Transfer the virtual machine’s active memory and precise execution state over a high-speed network (vMotion network), allowing the virtual machine to switch from running on the source vSphere host to the destination vSphere host. This entire process could take less than two seconds on a gigabit Ethernet network depending on the workload.

VMware vSphere® 5.1 vMotion presents a unified and integrated migration architecture that migrates live virtual machines, including their memory and storage, between vSphere hosts without any requirement for shared storage. When a virtual machine needs to be migrated between hosts that do not share storage, vMotion copies the data across the vMotion network using some of the Storage vMotion code. This shared-nothing live migration feature offers administrators significantly more simplicity and flexibility in managing and moving virtual machines across their virtual infrastructures compared to the traditional vMotion and Storage vMotion migration solutions.

Prior to 5.1, vMotion and Storage vMotion technologies worked together to provide the ability to migrate the entire state of the virtual machine, including its memory and storage, across vSphere hosts. However, this live-migration solution was limited to the hosts that shared a common set of datastores. In addition, migration of an entire virtual machine required two separate operations, for instance vMotion followed by Storage vMotion, or vice versa.

 2. Configuration

2.1   Enable vMotion

Enable vMotion functionality of the virtual switch through the vSphere client for each host:

2.2   Set up shared datastore (prior to vSphere 5.1)

The entire state of a virtual machine is encapsulated by a set of files stored on shared storage such as Fibre Channel oriSCSI Storage Area Network (SAN) or Network Attached Storage (NAS). As a cluster file system, VMFS lets multiple ESX/ESXi hosts access the same VMFS datastore concurrently. VMFS provides distributed locking for your virtual machine files, so that your virtual machines can operate safely in a SAN environment where multiple ESX/ESXi hosts share the same VMFS datastore.

Disk files for a virtual machine are not relocated when it is migrated using vMotion. Any RDMs remain as RDMs when the virtual machine is registered to another host. That is, no changes to the virtual machine itself are made.

The actual setup work simply means some LUN masking on array side to make devices visible to the ESX hosts and rescanning of the storage adapters. vSphere automatically discovers out the datastores on the newly detected LUNs.

For RDM (either physical or virtual mode) devices, make sure they have the same exact LUN id (address) to each of the hosts sharing the same devices, otherwise the migration will report the following error due to the mismatched VML identifiers (see VMware KB 1016210):   Virtual Disk ‘X’ is a mapped direct access LUN that is not accessible

3. How it works

Ref:  VMware vSphere® 5.1 vMotion Architecture, Performance and Best Practices

The active memory and precise execution state of the virtual machine is rapidly transferred over a high speed network (vMotion network), allowing the virtual machine to instantaneously switch from running on the source ESX host to the destination ESX host.

3.1 Migration of Virtual Machine’s Memory

http://blogs.vmware.com/vsphere/2011/02/vmotion-whats-going-on-under-the-covers.html

3.1.1 vMotion steps (iterative memory copy approach, aka “Pre-copy”), at a high level (vSphere 4.1):

1).    Shadow VM created on the destination host.
2).    Copy each memory page from the source to the destination via the vMotion network.  This is known as preCopy.
3).    Perform another pass over the VM’s memory, copying any pages that changed during the last preCopy iteration.
4).    Continue this iterative memory copying until no changed pages (outstanding to be-copied pages) remain.
5).    Stun the VM on the source and resume it on the destination.

 

3.1.2 Unified vMotion of memory in vSphere 5.1: (Insight Into vMotion: Architectures, Performance, Best Practices, and Futures)

Both Traditional vMotion and vMotion in vSphere 5.1 use the same iterative copy approach (aka memory “Pre-copy”)

• Trace and copy the memory in an iterative fashion
• Copy the modified memory during subsequent iterations

vMotion in vSphere 5.1 does more than Traditional vMotion

• Needs to handle ongoing IO writes to disk
• Memory Copy process works concurrently with IO Mirroring process (after bulk copy process)
• Both memory and disk changes are sent on same vMotion network

3.1.3 Atomic switch-over:

vSphere 5.1 vMotion guarantees atomic switch-over between source and destination hosts by ensuring both memory and disk state of the virtual machine are in lock-step before switch-over, and fails back to source host and source disks in the event of any unexpected failure during disk or memory copy.

Note that, to avoid overhead as much as possible, the storage network will be leveraged if both the source and destination host have access to the destination datastore. For instance, if a virtual machine resides on a local datastore and needs to be copied to a datastore located on a SAN, vMotion will use the storage network to which the source host is connected. In essence a Storage vMotion is used to avoid vMotion network utilization and additional host CPU cycles. Also if both the source and destination host are connected to the destination datastore and the datastore is located on an array that has VAAI enabled, Storage vMotion will offload the copy process to the array.

3.2 Migration of Virtual Device State

Like vMotion, vSphere 5.1 vMotion leverages the ability of vSphere to serialize the virtual device state of the virtual machine, which is generally less than 8MB in size. In some cases, it can grow in excess of 128MB, which can be transferred very quickly over a high-speed network.

3.3 Migration of External Network Connections

The vSphere virtual networking architecture makes it very easy to preserve existing networking connections even after a virtual machine is migrated to a different machine. Because each virtual Ethernet network interface card (vNIC) has its own MAC address that is independent of the physical NIC’s MAC address, the virtual machines can be migrated between the hosts and still keep the networking connections alive as long as both the source and destination hosts are on the same subnet. The migration is done efficiently so the networking connections do not time out. After the virtual machine is migrated, the destination ESXi host sends out a RARP packet to the physical network switch thereby ensuring that the switch updates its tables with the new switch port location of the migrated virtual machine. This migration is completely transparent to the remote clients of the virtual machine.

Note that http://frankdenneman.nl/2012/09/07/vsphere-5-1-vmotion-deepdive/ also provides some more details.

4. Convergence issue

In most circumstances the iterative copy works very well, assuming the host is able to transmit memory pages over the vMotion network faster than the VM can dirty new pages.

However, in the rare event that the VM is dirtying memory pages faster than vMotion can send them it is possible to get into as situation where the preCopy won’t be able to converge.

When the preCopy cannot converge, vMotion needs to decide whether to fail the vMotion or to proceed with switchover to the destination anyway.  It makes this decision by estimating the time required to transmit all the remaining outstanding pages.  By default, if this time is below 100 seconds vMotion will proceed with the switchover.  If it will take more than 100 seconds the vMotion will fail (timeout) with no impact on the VM.

In the event the VM passes the 100 second check, VMotion will stun the source and start running on the destination.  While the destination runs, the source will transmit the remaining pages to the destination using the “quick resume” capability introduced with vSphere 4.1.

The default 100 seconds can be tuned via vmotion.maxSwitchoverSeconds.

In summary:

• You should be able to vMotion any workload as long as it is dirtying memory pages at a rate that is less than your vMotion network transmit rate.
• vMotion will only transfer the VM to the destination if it is certain that it can complete the memory copy.
• If vMotion cannot complete the memory copy it will fail with no impact to the running VM.

4.1   Quick Resume (vSphere 4.1)

http://www.yellow-bricks.com/2011/04/13/vmotion-and-quick-resume/

Quick Resume is a feature that allows you to vMotion a virtual machine which has a high memory page change rate. Basically when the change rate of your memory pages exceeds the capabilities of your network infrastructure you could end up in a scenario where vMotioning a virtual machine would fail as the change rate would make a switch-over impossible. With Quick Resume this has changed.

Quick Resume enables the source virtual machines to be stunned while starting the destination virtual machine before all pages have copied. However, as the virtual machine is already running at the destination it could possibly attempt to touch (read or write) a page which hasn’t been copied yet. In that case Quick Resume requests the page from the source to allow the guest to complete the action while continuously copying the remaining memory pages until all pages are migrated. But what if the network would fail at that point, wouldn’t you end up with a destination virtual machine which cannot access certain memory pages anymore as they are “living” remotely? Just like Storage IO Control, vMotion leverages shared storage. A special file would be created in the case Quick Resume is used and this file is basically used as a backup buffer. In the case the network would fail this file would allow for the migration to complete. This file is typically in the order of just a couple MBs. Besides being used as a buffer for transferring the memory pages it also enables bi-directional communication between the two hosts allowing the vMotion to complete as though the network hadn’t failed. Is that cool or what?

The typical question that arises immediately  is if this will impact performance? It is good to realize that without Quick Resume vMotioning large memory active virtual machines would be difficult. The switch-over time could potentially be too large and lead to temporary loss of connection with the virtual machine. Although Quick Resume will impact performance when pages that are not copied yet are accessed, the benefits of being able to vMotion very large virtual machines with minimal impact by far outweigh this temporary increase of memory access time.

4.2 Stun During Page Send (SDPS, vSphere 5.0)

http://www.yellow-bricks.com/2011/07/20/vsphere-50-vmotion-enhancements/

Quick Resume is a vSphere 4.1 vMotion enhancement and now it is replaced with Stun During Page Send, or also often referred to as “Slowdown During Page Send” is a feature that “slowsd own” the vCPU of the virtual machine that is being vMotioned. Simply said, vMotion will track the rate at which the guest pages are changed, or as the engineers prefer to call it, “dirtied”. The rate at which this occurs is compared to the vMotion transmission rate. If the rate at which the pages are dirtied exceeds the transmission rate, the source vCPUs will be placed in a sleep state to decrease the rate at which pages are dirtied and to allow the vMotion process to complete. It is good to know that the vCPUs will only be put to sleep for a few milliseconds at a time at most. SDPS injects frequent, tiny sleeps, disrupting the virtual machine’s workload just enough to guarantee vMotion can keep up with the memory page change rate to allow for a successful and non-disruptive completion of the process. You could say that, thanks to SDPS, you can vMotion any type of workload regardless of how aggressive it is.

It is important to realize that SDPS only slows down a virtual machine in the cases where the memory page change rate would have previously caused a vMotion to fail.

This technology is also what enables the increase in accepted latency for long distance vMotion. Pre-vSphere 5.0, the maximum supported latency for vMotion was 5ms. As you can imagine, this restricted many customers from enabling cross-site clusters. As of vSphere 5.0, the maximum supported latency has been doubled to 10ms for environments using Enterprise Plus. This should allow more customers to enable DRS between sites when all the required infrastructure components are available like, for instance, shared storage.

4.3 Multi-NIC vMotion Support (vSphere 5.0)

http://www.yellow-bricks.com/2011/07/20/vsphere-50-vmotion-enhancements/

One of the most substantial and visible changes is multi-NIC vMotion capabilities. vMotion is now capable of using multiple NICs concurrently to decrease the amount of time a vMotion takes. That means that even a single vMotion can leverage all of the configured vMotion NICs. Prior vSphere 5.0, only a single NIC was used for a vMotion enabled VMkernel. Enabling multiple NICs for your vMotion enabled VMkernel’s will remove some of the constraints from a bandwidth/throughput perspective that are associated with large and memory active virtual machines. The following list shows the currently supported maximum number of NICs for multi-NIC vMotion:

• 1GbE – 16 NICs supported
• 10GbE – 4 NICs supported

5. Metro vMotion

http://www.yellow-bricks.com/2011/08/03/vsphere-5-metro-vmotion/

6. Configuration Requirements

A number of specific virtual machine configurations can prevent migration of a virtual machine with vMotion. The following virtual machine configurations can prevent migration with vMotion:

• You cannot use migration with vMotion to migrate virtual machines using raw disks for clustering purposes.
• You cannot use migration with vMotion to migrate a virtual machine that uses a virtual device backed by a device that is not accessible on the destination host. (For example, you cannot migrate a virtual machine with a CD drive backed by the physical CD drive on the source host.) Disconnect these devices before migrating the virtual machine.
• You cannot use migration with vMotion to migrate a virtual machine that uses a virtual device backed by a device on the client computer. Disconnect these devices before migrating the virtual machine.
• You cannot vMotion across datacenters even managed by a single vCenter Server.

7. vMotion futures

• vMotion across vCenter Servers
• vMotion across VDC (vCloud Dirctor Virtual Datacenters)
• Long-distance vMotion (between datacenters)

VMware VM Clone

1. VM Clone Overview

 A clone is a copy of an existing virtual machine. The existing virtual machine is called the parent of the clone. When the cloning operation is complete, the clone is a separate virtual machine — though it may share virtual disks with the parent virtual machine according to clone type).

• Changes made to a clone do not affect the parent virtual machine. Changes made to the parent virtual machine do not appear in a clone.
• A clone’s MAC address and UUID are different from those of the parent virtual machine.

If you want to save the current state of the virtual machine, so you can revert to that state in case you make a mistake, take a snapshot. If you want to make a copy of a virtual machine for separate use, create a clone.

Cloning a virtual machine creates a point-in-time duplicate of the virtual machine with the same configuration and installed software as the original.

Optionally, you can customize the guest operating system of the clone or use a customization specification to change the virtual machine name, network settings, and other properties if VMware Tools is installed in the virtual machine.

There are two types of clone:

• A full clone is an independent copy of a virtual machine that shares nothing with the parent virtual machine after the cloning operation. Ongoing operation of a full clone is entirely separate from the parent virtual machine.
• A linked clone is a duplicate of a virtual machine that uses the same base disk as the original, with a chain of delta disks to track the differences between the original and the clone. Actually, a linked clone is a writable snapshot. This conserves disk space, and allows multiple virtual machines to use the same software installation.

Linked clone cannot be created, configured or deployed from the vSphere client or ESXi CLI, they do not get a great deal of exposure. vCloud Director and VDI now use linked clones for faster deployments of VMs, as well as saving disk space by avoiding complete clones of VMs from a vApp Template.

 2. Taking Clones

2.1 User Configuration Steps:

• Enter a virtual machine name
• Select a location (datacenter and/or folder)
• Select a host or cluster on which to run the new virtual machine
• Select a resource pool in which to run the virtual machine
• Select the datastore location where you want to store the virtual machine files
• Optionally, select the format for the virtual machine’s disks (default is the same as source; each virtual disk can be individually customized through the advanced configuration button)
• Optionally, select a guest operating system customization option (VMware Tools should be installed)
• Optionally, select whether to power on the virtual machine or edit virtual machine settings

2.2 vSphere Working Steps:

• Creating snapshot of Virtual Machine
• Copying Virtual Machine files
• Creating the new Virtual Machine
• Powering on Virtual Machine (if user selected to power on the VM after cloning)

 3.    Application-consistent Clone

By default, the cloning operation firstly creates a quiesced application-consistent snapshot and then performs the copy to achieve an application-consistent clone.

When heavily loaded applications, such as load generators, are running in the guest operating system during a clone operation, the virtual machine quiesce operation can fail and VMware Tools might be denied CPU resources and time out. So if a load generator is running in the virtual machine, stop it before you perform the clone operation.

When creating a hot clones of VMs that have high I/O you may encounter the following error (http://www.vsysad.com/2013/02/high-io-on-vm-causes-clone-to-fail/):

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

This is s common issue when trying to perform clones of very busy production servers such as SQL Server or Exchange Server. As these applications perform constant reads/writes the usual quiescing function (when initiating a clone or snapshot  fails to flush all of the operations to disk.

A quick and dirty fix if you simply want to clone the VM is to stop the VMware Tools Service on the guest and then re-run the clone. It should complete OK. This basically bypasses the quiescing of the file system and produces a crash-consistent clone.

VMware KB article 1018194 highlights this issue and prescribes various fixes depending on your circumstance.

4.    Cloning VM using vStorage APIs for Array Integration (VAAI)

 VMware’s Storage APIs for Array Integration (VAAI) is an API for storage partners to leverage that permits certain functions to be delegated/offloaded to the storage array, thus greatly enhancing the performance of those functions.

It is the hardware-accelerated Full Copy primitive defined in VAAI that enables the offloading of VMware (Full) clones to the storage array.

In addition to deploying new virtual machines from a template or through cloning, Full Copy is also utilized when doing a Storage vMotion. When a virtual machine is migrated between datastores on the same array the live copy is performed entirely on the array.

The VAAIs are enabled (Hardware Acceleration) by default on both the storage arrays and ESX server.

 

5.    Cloning across datacenters

http://www.rayheffer.com/300/vmware-vsphere-virtual-machine-live-clones-across-datacenters/

Prior to VMware ESX 4 (vSphere) it was not possible to clone a running virtual machine to another datacenter, or to a virtual machine data store that wasn’t available on the source host. Now with VMware vSphere it is possible to clone a running virtual machine to another datacenter, even if the destination data store is not presented to the source host.

How does this work when the target hosts volume isn’t even accessible on the originating server?

The answer is in the Service Console network. When source and destination disk volumes are present, then it will initiate a disk to disk transfer (over iSCSI or fibre channel). However, when the target storage is unavailable it will now use the Service Console network to transfer the data. Magic!

This is an excellent feature, as prior to ESX 4 I had to temporarily present the target volume to the source host during the migration or cloning process. Now I can keep storage between my datacenters seperate.

Why the Service Console Network?

Providing the service console network is used for management only, this is the best option. You certainly wouldn’t want the VMkernal network to be saturated with traffic generated from a storage migration, as this would impact VMotion.

6.    Cloning VM with RDMs

For physical mode RDMs: Cloning of VMs that use physical mode RDMs is not supported;  For virtual mode RDMs:    Cloning of VMs that use virtual mode RDMs will convert RDMs into VMFS virtual disks (regardless if you choose to change the format or not) and RDM contents will be copied into the virtual disks.

 

Reference:

• vSphere Virtual Machine Administration

VMware VM Snapshot

1. VM Snapshot Overview

Snapshots preserve the state and data of a virtual machine at the time you take the snapshot. Snapshots are useful when you must revert repeatedly to the same virtual machine state, but you do not want to create multiple virtual machines.

A VM snapshot preserves the following information:

• Virtual machine settings. The virtual machine directory, which includes disks that were added or changed after you took the snapshot.
• Power state. The virtual machine can be powered on, powered off, or suspended.
• Disk state. State of all the virtual machine’s virtual disks.
• (Optional) Memory state. The contents of the virtual machine’s memory.

2. Taking Snapshots

Taking a snapshot preserves the disk state at a specific time by creating a series of delta disks for each attached virtual disk or virtual RDM and optionally preserves the memory and power state by creating a memory file.

Two options when you could select while taking a snapshot:

• snapshot the virtual machines’s memory   –   to get a live snapshot
• quiesce guest file system (Need VMware Tools installed on guest system)  – to get a application consistency (vs. crash consistency) snapshot

How consistency snapshot (quiescing guest file system) works:

http://wiki.answers.com/Q/How_does_the_VMware_FileSystem_Sync_Driver_vmsync_work

The VMware FileSystem Sync Driver (vmsync) is a new feature that creates backups of virtual machines. Creating quiesced snapshots using the SYNC driver requires waiting for I/O to
drain in the guest operating system. This might affect time sensitive applications, such as DBHammer.

If you do not install the SYNC driver when installing VMware Tools, you can avoid
the delay caused by the I/O draining. However, it results in snapshots being only
crash-consistent, unless you provide custom quiescing through pre-backup and
post-backup scripts in the guest

If the guest operating system is Windows Vista, Windows Server 2003, or other
newer Windows operating systems, a Volume Shadow Copy Services (VSS)
module is installed. For other, older Windows operating systems, the
Filesystem Sync driver is installed. These modules enable backup applications
to create application-consistent snapshots. During the snapshotting process,
certain processes are paused and virtual machine disks are quiesced.

3. Revert (goto) to a snapshot

After you take a snapshot, you can revert to a snapshot from the virtual machine’s right-click Snapshot Manager menu to restore that snapshot at any time.

If you revert to a snapshot that didn’t preserve the the memory (select “snapshot the virtual machines’s memory”) , it’ll end up becoming the power-off state.

4. Snapshot Limitations:

• VMware does not support snapshots of raw disks, RDM physical mode disks, or guest operating systems that use an iSCSI initiator in the guest.
• Virtual machines with independent disks must be powered off before you take a snapshot. Snapshots of powered-on or suspended virtual machines with independent disks are not supported.
• Each branch in a snapshot hierarchy tree can have up to 32 snapshots.

Reference:

• vSphere Virtual Machine Administration
• VMware KB 1015180: Understanding virtual machine snapshots in VMware ESXi and ESX
• http://searchvmware.techtarget.com/tip/How-VMware-snapshots-work