Storage I/O control for Hyper-V

Issues related to the storage layer are the number one cause of performance issues on virtualization platforms like VMware vSphere and Microsoft Hyper-V.
Sooner or later any storage system will suffer from bad performance. While disk capacity is mostly monitored, the latency and throughput of the storage layer are not. All of a sudden latencies reach a level which results in a complete saturation of the system. This could be because the number of virtual machines is increasing over time, or by a sudden burst of storage i/o

So as an administrator you want control over storage performance. Control of storage I/O is especially important for multi-tenant infrastructures. As a provider you do not want  for instance a database query run by one of the customers result in a degraded performance for all other tenants of the shared infrastructure. Control of storage I/O means that a proces running on a virtual machine cannot consume all the available storage resources anymore but is limited.

This posting will focus on the methods to control storage i/o for Hyper-V when HP EVA and Qlogic fiberchannel HBA are used. The information can also be used for HBA’s and storage array’s of other vendors.

I like to thank HP for all the information I got while working with them. A lot of the information in this article cannot be found anywhere else on Internet. 

Mind there is no best practice for tuning the storage i/o. There are so many components involved in the storage system which has an effect on the total performance there is no best practice.

Unlike VMware vSphere which has a feature called Storage I/O Control, Hyper-V does not have any built in mechanism to dynamically or even statically control storage I/O.

Melio FS (third party) could be a solution to have control of storage i/o. It has a quality of  service feature which enables to set priorities per VM. See this posting for more information.

If you need more storage power, the HP IO Accelerator cards for HP Blades might be a (expensive) option. See this for more information. Microsoft used these cards for the MMS2011 lab environment. Taken from the Microsoft site:

One of the key elements of this solution were the HP IO Accelerator cards which provide high read and write performance: up to 100K IOPS and throughput of 800MB/s read and 600MB/sec write. These high performance IO cards were installed on each of the 32 Blade servers. The Differencing Disks for Virtual Machines are stored on these cards while the shared Parent VHD is stored on the SAN. This allows us to rapidly tear up and tear down an entire lab environment within minutes

 Without Melio FS you are stuck with adjusting  HBA firmware  settings and driver settings in the Windows registry.
I guess it can be best descriped as follows: you do not want to have congestion on a motorway at all times. So at the entrance ramp traffic lights are operational to control the numbers of cars per minute entering the motorway. However, the traffic lights are either active or not active. As we want control, the traffic lights are always active, even when there is no congestion on the motorway. 
Using Melio or VMware SIOC  the traffic light are smart enough to be disabled outside rush hours.

Four configuration settings are available for Qlogic HBA’s to control storage I/O on Windows Server/Hyper-V. These settings are very poorly documented. When doing research I could not find much info.

1. Adjusting the HBA transfer size.  
2. Adjusting the HBA queue depth or execution throttle as Qlogic names it
3. Adjusting the HBA software driver qd setting
4. Adjusting the HBA driver throttle settings

1. Adjusting the HBA tranfer size
The most important setting which will have an immediate effect on latency during congestion is the HBA datatransfer size. The term HBA block size is often used as well. This setting is default set at 512KB. This is the maximum transfer size HP EVA can accept. However the 512KB setting resulted under certain conditions in an overloaded EVA storage showing high latencies and degraded performance. When set to 128KB we saw the same issues on performance. When the Qlogic HBA was set at a transfer size of 64 KB the EVA could not be overloaded anymore.
Only since the end of 2010 the ability to decrease the transfer size was added to the HBA’s of Emulex, Qlogic and Brocade.

For Qlogic HBA’s the command line tool needs to be used to set the transfer size. To set it at 64KB use the following command.

c:\>qlfc -tsize /fc /set 64

See the HP document titled HP StorageWorks Enterprise Virtual Array transfer size tuning option with latest HBA drivers on Windows for more information.

2. Adjusting the HBA queue depth
The queue depth is a setting on firmware of the HBA card. Qlogic names this Execution Throttle. Taken from the qlogic website:  
The HBA Execution Throttle is set to 16 by default.  It specifies the maximum number of simultaneous commands the adapter will send on any one target port. The default value is 16, increasing the value above 64 has little to no effect, because the maximum parallel execution of SCSI operations is 64.

The lowest possible value of 1 will result on very low iops and very high latency.

Execution Throttle for Qlogic HBA’s can be set using SAN Surfer software. Select the host, select each of the ports.Select the parameters tab, select the Advanced HBA Parameters option. Set Exection Throttle and click on save. Repeat for the other port. A reboot is not needed.

A queue exist on the storage array controller port as well, this is called the “Target Port Queue Depth“. Modern midrange storage arrays, like most EMC- and HP arrays can handle around 2048 outstanding IO’s.
When a target port’s Queue Depth is reached, the system will not issue any new commands until one of the current commands finishes executing.

To increase the number of outstanding commands to the target port, increase the Execution Throttle.
WARNING:  Increasing the Execution Throttle may cause SCSI timeouts.

BEST PRACTICE:  Divide the target port’s maximum outstanding commands (queue) by the number of host ports that access the target port. 

(Storage Port Max Queue)/(Host Ports)=Execution Throttle

Example:
A storage port has a maximum queue of 2048 divided by 8 dual-ported servers (16 host ports).  In this case, the Execution Throttle is set to 128 on all the host HBA ports.

What is the relationship of queue depth  (qd) in the Windows registry and the execution throttle in the HBA and what are some examples?
http://kb.qlogic.com/KanisaPlatform/Publishing/436/2034168_f.SAL_Public.html

3. Adjusting the HBA drive throttle setting

The Qlogic driver configuration settings can be set in the Windows registry. Two parameters can be set: qd and throttle.
The exact effect of changing those two settings are unclear (because undocumented) but they do have an effect. 

Select HKEY_LOCAL_MACHINE and follow the tree structure down to the QLogic driver as follows:

            HKEY_LOCAL_MACHINE
              SYSTEM
                CurrentControlSet
                  Services
                    Ql2300
                      Parameters
                        Device

Double click on

           DriverParameters:REG_SZ:UseSameNN=0
   
-If the string “throttle=m” does not exist, append to end of string

           ;throttle=n

   where n is the value you desire.

– If the string “throttle=m” exists, change the value to the desired value.
– Click OK.
– Exit the Registry Editor, then shutdown and *reboot* the system.

4. Adjusting the HBA driver queue depth setting
The same registry key listed above can be expanded with ;qd=<value>

I assume the QD setting determines how many commands the HBA is willing to accept and process sent by Hyper-V. A QD=1  will result in low number of iops and high latency. While qd=16 will have much more iops.

Advertisements

About Marcel van den Berg
I am a technical consultant with a strong focus on server virtualization, desktop virtualization, cloud computing and business continuity/disaster recovery.

One Response to Storage I/O control for Hyper-V

  1. deinoscloud says:

    Very interesting post…

    I have another formula for you in regards to outstanding SCSI IO:

    T => P*q*L

    Where:
    „ P = The number of host paths connected to that array device port.
    „ q = Queue depth per LUN on the hosts (for the host port); i.e., the maximum I/Os outstanding per
    LUN from the host at any given time.
    „ L = The number of LUNs configured on the array device port as seen by a particular host path.
    „ T = The maximum queue depth per array device port, which signifies the maximum number of I/Os
    outstanding (that can be handled) on that port at a given time. The value for T is typically 2048 for
    EMC arrays.
    „ Q = Queue depth per HBA port; i.e., the maximum I/Os that an HBA can have outstanding at an
    instant. This parameter is needed only if q (queue depth per LUN) is not available.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: