19 controllers removed from VMware Virtual SAN (VSAN)compatibility list

VMware announced in this KB article 19 discontrollers initially listed on the VSAN compatibility list have been removed from this list. This makes the controllers unsupported for usage in VSAN configurations.

The reason for this removal is according to VMware’s KB article:

As part of VMware’s ongoing testing and certification efforts on Virtual SAN compatible hardware, VMware has decided to remove these controllers from the Virtual SAN compatibility list. While fully functional, these controllers offer too low IO throughput to sustain the performance requirements of most VMware environments. Because of the low queue depth offered by these controllers, even a moderate IO rate could result in IO operations timing out, especially during disk rebuild operations. In this event, the controller may be unable to cope with both a rebuild activity and running Virtual Machine IO causing elongated rebuild time and slow application responsiveness. To avoid issues, such as the one described above, VMware is removing these controllers from the Hardware Compatibility List.

 

One of the controlers removed is the Dell PERC H310. Likely one of the reasons this controller has been deleted is a serious issue described at Reddit.com. A VSAN user experienced basically a meltdown of its virtual infrastructure when a VSAN node failed and after a while a rebuild started. This caused so many IO the PERC H310 could not handle and all VMs came to a standstill.

These are all controllers removed from the VSAN compatibility list

vsan-controllers

VMware release VMware Workstation 10.0.0 fixes OpenSSL

VMware released VMware Workstation 10.0.3 at July 1. This new build fixes OpenSSL issues.

VMware Workstation 10.0.3 has been updated to the OPENSSL library version openssl-0.9.8za where necessary to address CVE-2014-0224, CVE-2014-0198, CVE-2010-5298, and CVE-2014-3470.

 

Download VMware Workstation 10.0.3 Windows or for Linux here.

VMware opens public beta for next version of vSphere

For the first time ever VMware opens a beta program of vSphere available for the public. So far participating in the beta was only available for VMware partners, vExperts and other selected groups.

Now everyone can participate and have a look into the future of vSphere. The next release will most likely be vSphere 6.0 although VMware never communicates over version numbers before product announcement.

Joining is very simple by filling in name, role, company and country. Then digital sign and you are in the beta within minutes.

Please remember that this vSphere Beta is private even though it is open to everyone. Do not share information from this Beta Program with those not in the Beta Program. What you learn and see in this Beta is meant to be kept confidential per the Master Software Beta Test Agreement and Program Rules.

The ground rules of the program are here.

This is an interesting change in policy of VMware. Microsoft always has been very open about new versions of for example Hyper-V and enabling the public to download beta’s.  VMware preferred to keep features and even the release number of vSphere secret tilll unveiled at VMworld.

I am not sure about the reason for this change in policy. Likely VMware made the beta open to public for quality reasons to make sure as many scenario’s are tested and teething problems are resolved before the product reaches GA. VMware Virtual SAN has been running the beta for a relative long time for the same reason.

Another , less likely, reason could be VMware wants to be more open on the future of products for marketing reasons.

The VMware blog announcing the beta is here

Join the beta here.

I will be joining PQR at July 1!

I am very excited to announce I will be joining PQR at July 1. PQR is well known for its knowledge on many infrastructure solutions, services and highly qualified and passionate staff.

As a senior technical consultant I will continue to focus on solutions and services in datacenters. Mainly VMware and Microsoft. I will be the sixth VMware vExpert working for the company.

I also just signed a contract with VMware to author blogposts on a regular base. These posts  which will be published on the VMware SMB blog.

Exciting and busy  times ahead as my book on Microsoft Cloud OS is almost ready. Working hard on the chapter which provides insight on Azure Site Recovery service.

Looking for a cost effective way to manage Hyper-V? Introducing 5nine Manager 5.0 for Hyper-V

For managing Hyper-V servers, virtual networks, storage and virtual machines customers are required to buy the complete Microsoft System Center Suite. System Center Virtual Machine Manager (SCVMM), which is used for management of Hyper-V, is not available as a separate product. Customers have to buy the complete suite including backup, monitoring and deployment tools.

For small organizations the costs of System Center can be unattractive especially because they get functionality they probably either already have or will not need. Not every organization needs Configuration Manager or Operations Manager.

5nine Manager 5.0 for Hyper-V is a nice alternative to buying the full blown System Center suite . Version 5 has been released end of June. Even if Hyper-V is not used this product is usefull. If Windows Server Core (no GUI) is used 5nine Manager offers a GUI to manage those servers.

The product has some unique features like the ability to manage the Hyper-V role installed on Windows 8 and Windows 8.1 which Microsoft Hyper-V Manager nor SCVMM is able to.

5nine-hyperv-5-2

Another very nice feature is the ability to manage the free Microsoft Hyper-V Server. This software once installed does not come with a graphical userinterface. Basic configuration can be done in a command prompt. Advanced configuration needs to be performed using tools like PowerShell.  5nine Manager offers a graphical userinterface for Hyper-V installs without a GUI.

It is available in three editions:

  1. 5nine Manager free edition
  2. 5nine Manager
  3. 5nine Manager with antivirus.

The free edition only allows to do basic management. Actions like performing a Live Migration or a Storage Migration are not enabled in the free edition. However it offers some features not available in Microsoft Hyper-V manager console.
5nine Manager commercial edition has all features you want as an administrator.
5nine Manager with antivirus has all features of 5nine Manager including agentless antivirus for virtual machines.

A feature compare can be seen here.

What’s new in 5nine Manager 5.0 for Hyper-V

  • Automated virtual machine (VM) provisioning
  • Enhanced cluster management
  • VMs Guest connection views through FreeRDP or Microsoft controls
  • Support in the GUI for configuration of Hyper-V Replica which enables failover of production workloads to a secondary site for disaster recovery
  • Group operations for multiple VMs
  • Applications Logs

Additionally, 5nine Manager for Hyper-V performs the following features that are absent in the
standard MS Hyper-V management tool:

  • Own graphical user-friendly interface – file manager with built-in transfer virtual hard drive with a capacity up to 127 Gb for exploring files and network shares, even on Windows Core and Free Hyper-V.
  • Quality of Service management – setting of minimum and maximum Input Output operations per second (IOPS) throttling for the virtual hard disk attached to a virtual machine (applies for hosts with Windows Server 2012 R2)
  • System Status Report presented in the intuitive graphical form. Available in the full version of 5nine Manager for Hyper-V.
  • Failover cluster manager function, such as VM migration between nodes. Available in the full version of 5nine Manager for Hyper-V.

qualityofservice

5nine Manager 5.0 for Hyper-V is available immediately. It is priced per host, and pricing begins at $199

The administrators guide (for version 4.2) can be found here.

Using Office 365 and DirSync? Make sure to license all sync-ed Active Directory accounts !

Office 365 is used by many organizations to offer mail and calender functionality to users. A common scenario is the usage of Office 365 in retail. Staff in shops just have access to Office 365.

IT can make two choices for management of user accounts for Office 365 users.

  1. create user accounts in Azure Active Directory (AAD)
  2. create user accounts in on-premises Active Directory and sync those to AAD.

The advantage of  option 2 is central management. There is just a single directory to manage. Another advantage is that users which need to authenticate to AD can use a single useraccount and password.

To synchronize user accounts and passwords stored in on-premises AD to AAD Microsoft offers a free tool called DirSync.

Mind that for each user account a Windows Server Client Access License (CAL) is required! The cost for such a CAL is about Euro 30,- or $ 40,- per user or device

 

Microsoft licensing explained (a try)

Microsoft licensing is hard and complex. Many options are offered by Microsoft for the right to use their software.

It is a skill to be able to provide the best advise and to implement licensing the correct way. First there are many different agreements, plans and programs. Secondly there are many rules, small letters, exceptions and not so obvious restrictions documented in not always very well known documents.

In this blog I will provide a highlevel overview of Microsoft licensing. When you understand the basics, Microsoft’s  detailed information on all the agreements and enrollments will provide the details.

Buy or lease agreements

The right to use Microsoft software can be obtained via two ways:

  1. as a retail boxed software. Customers buy a box with the software on DVD and the license to use the software.
  2. as part of Volume Licensing. Customers receive a license. Software is downloaded from a Microsoft portal. Customers which agree in a Volume Licensing agreement get discount and other benefits.

Volume Licensing  can be either purchased, leased or leased/purchased. Depending cash position, dynamics of the organization a lease or purchase is the best option.  Agreements for organizations with over 5 computers are part of the Microsoft Volume Licensing. There are programs for organizations with 5-250 computers and for organizations with over 250 computers.

For Service Providers Microsoft has a pay-as-you-go model available. A service provider is only charged for the actual usage of a product. This can even be a single hour.

Volume Licensing has three important components which you should be aware of.

  • agreements or programs
  • enrollments
  • software assurance

An agreement is an procurement contract. In the contract Microsoft and the customer agree on many things. It has information on processes like True-Up and defines the length of the contract, the payment, term for additional product license acquisitions, subsequent orders and perpetual rights.

Enrollements are available as part of enterprise agreement and select licensing programs. Enrollments are contracts about the ordering of software. The customer promises to buy a certain quantity of software and certain types of software. In return the customer receives discount.

Enrollments are a Microsoft way to push customers to a certain direction. This can be towards purchasing Software Assurance or towards purchasing Hyper-V, standardize on Office or consume cloud services like Azure.

Software Assurance (SA) is a kind of maintenance contract. Customers have the right to use the latest version software. Another benefit is the right to move license to other servers or to cloud environments. Microsoft is constantly moving benefits which were included in the  license to SA. Cost of SA is about 25% of the license price paid each year.

For some agreements the customer can decide to buy SA as addition to the license. In many agreements SA is included. SA has been introduced by Microsoft to guarantee a steady cashflow. Before SA Microsoft got a lot of revenue after the release of a new version of software. Then the revenue dropped to be increased at the next release. With SA Microsoft monthly gets money from customers on SA.

The reference guide explains it all.

Available Agreements 

The image below shows the major agreements available to corporate . There are also agreements for government, education, charity, service providers and partners of Microsoft.

These programms are all focussed on on-premises software usage. They offer however the possibility to use Microsoft cloud services as well.

Pricing for all agreements are very different. Organizations with over 250 users/computers get a better price than those with a few users. Organizations with many desktops get better discounts than with few desktops. Schools get about 80 % discount for licenses.

To be able to start an agreement organizations must meet certain conditions. For example a minimum purchase or have all workstations installed with the latest Office version.

It is perfecty possible to have a customer use various agreements.

Microsoft license agreements

Agreements for small orgs with 5-250 computers or users 

  • Open License
  • Open Value
  • Open Value Subscription

Open License is just buying the license. This provides the customer the right to use to software as long as he wants. There is no rental available. A maitenance program called Software Assurance can be purchased if the customer wants to upgrade to the most recent version software. Software is paid upfront.

Open Value is including Software Assurance. The customer can annualize their billing over 3 years or they can pay upfront. After three year no more payments and the license belongs to the customer. Licenses can only be added to the agreement. So if your organization shrinks you might have to pay for more licenses than needed.

Open Value Subscription means the customer rents the software. When the rental period has expired and is not extended the software may not be used anymore. Licenses can be added (True Up) or removed from the agreement (True down)

These agreements can be sold by any Microsoft partner.

Agreements for large orgs with over 250 computers or users

  • Select and Select Plus
  • Enterprise Agreement
  • Enterprise Agreement Subscription

Select Plus is a purchase programm with an option to buy SA. To qualifiy for Select Plus customers have to buy an initial order of 500 points per product pool. Microsoft products are groups in three pools: applications, systems, and server.  500 points equals for example 250 Office licenses. The more points a customer has, the better discount.  More on Select Plus here.

Select Plus is being replaced by  a new agreement for midsized organisations called Microsoft Products and Services Agreement (MPSA). A FAQ here. The general rollout of MPSA licensing is planned for late 2014 which is when all enrollments will be expected to start following the MPSA model. Microsoft describes the MPSA as blending elements of the Microsoft Business and Services Agreement, Select Plus licensing and Microsoft Online Services subscriptions. A compare between Select Plus and MPSA is here.

Microsoft Enterprise Agreement can be purchased at a Microsoft Licensing Solution Provider  only. The Microsoft Enterprise Agreement allows organizations with more than 250 PCs, devices and/or users to purchase  Microsoft licenses  and cloud services over a three-year period at the best available pricing. As Software Assurance is included customers have the right to use the most recent version. At the end of the three years the license becomes a perpetual license. This means the customer can use it as long as he wants. Customers cannot reduce the number of licensed software during the three year period.

Microsoft Enterprise Subscription Agreement is a pay-as-you-go model. It is a rental agreement for three year which includes Software Assurance. Customers have the ability to decrease the number of licensed software each year during the True-Up process. There is no initial CAPEX. Costs are booked as operational costs. Licenses in this agreement are  nonperpetual (or subscription) licenses, which provide the right to use a particular licensed product until the end of the license-agreement term.  A requirement is that all desktops of the customer are standardized on Office, Windows or the Core CAL.

More info here. 

The difference between both is explained here.

MOSA Microsoft Online Subscription Agreement

MOSA is a dedicated licensing agreement for using Microsoft Online Services like Office 365. After signing the MOSA customers can purchase subscriptions under the  Microsoft Online Subscription Program (MOSP). This is a subscription-based Microsoft Volume Licensing program for organizations with one or more users that want to subscribe to, activate, provision, and maintain services seamlessly and affordably. The services available in this program  include Office 365, Windows Azure Platform, CRM Online, and Intune

Agreements for Service Providers

  • Microsoft Services Provider License Agreement (SPLA)

Agreements for Education

  • School Enrollment
  • Campus and School Agreement (CASA) = Specifically for qualified academic institutions such as schools, colleges, universities, including research facilities, interested in purchasing five or more licenses.
  • Enrollment for Education Solutions (CASA+EES or OVS-ES).
  • Microsoft Campus Agreement for higher education
  • Microsoft School Agreement for primary and secondary education institutions.

Enrollments

An enrollement is an extension of some agreements like the Enterprise Agreement. The target of an enrollment is to stimulate customers to buy a number of licenses or standardize on Microsoft products. In return they get discount.

For Enterprise Agreements the following enrollements are available:

  • Server and Cloud Enrollment (SCE). The most recent enrollment which replaces the ones listed below.
  • Enrollment for Application Platform (EAP)
  • Enrollment for Core Infrastructure  (ECI)
  • Enrollment for Windows Azure (EWA)

Server and Cloud Enrollment (SCE) covers datacenter products both on-premises and in the cloud. It  includes SQL Server, SharePoint, Biztalk, the Core Infrastructure Suite, Visual Studio and Azure.  There is no growth of licenses required but a minimum purchase of licenses is required. For example to meet SCE requirements customers must purchase minimum of 50 SQL Server core licenses or 5 SharePoint licenses. Also customers needs to cover the entire Enterprise server footprint with SA.
More info on SCE

Enrollment for Application Platform contains SQL Server, Visual Studio, SharePoint and Biztalk. It provides up to 40% discount but a license growth is expected to be 20-30% over a baseline.

Enrollment for Core Infrastructure offers discounts for a bundle of  Windows Server and System Center. Two bundles are available. The Core Infrastructure Suite Datacenter and Core Infrastructure Suite Standard. Discount of 20% is given on bundle as compared to product licenses individually

More info on ECI and EAP

This image compiled by Enpointe.com clearly shows the differences in enrollements

enrollment programs

Legal documents

When the agreement has been signed the customer should comply to certain Microsoft published documents. These documents explain how licenses should be applied.

The Microsoft Business and Service Agreement (MBSA) is the perpetual umbrella agreement that dictates the basic terms for all signed licensing agreements between your organization and Microsoft. This master agreement defines contract terms common to Microsoft licensing, service, and support agreements. MBSA is applicable for Select, Select Plus and Enterprise Agreements.

An important document for on-premises software is the Product Usage Rights. Service Providers have to use the Services Provider Use Rights (SPUR)  document. These are frequently published by Microsoft. The PUR describes per Microsoft product how the license should be used.

The Product List is another important document. It has info on avilability of new products, point values, product migration paths and Software Assurance benefits.

Both are binding documents. The customer should apply to what is written in those documents.

It is important to understand to what license the PUR applies. An excellent post at microsoftlicensereview.com about the PUR states:

  • For customers that elect to leverage downgrade rights, the Product Use Rights for the version licensed, not the version running will apply.

 

Some terms being used

Microsoft uses various terms in licensing documents. Some are explained here.

True-up
Customers with an Enterprise Agreement or Open Value contract agree to use a certain number of licenses. The customer is allowed to use more licenses than agreed. Once a year the customer reports the number of licenses in use. The difference between the number of licenses in the EA or Open Value and the actual usage is the True-up. Reporting the true-up once a year is mandatory.

Level and No Level
Microsoft provides discount when customers purchase a certain number of licenses. If the number of license is below a threshold this is called no Level. Level means the customer has acquired more licenses than the threshold and will get more discount. So LEVEL licenses are always cheaper than NOLEVEL

Levels in some products have numbers like Level C or Level D.

Pricing levels 
For enterprise Agreements there are pricing levels. Starting at Level A which provides the lowest discount to Level D which offers the highest discount. Governments always get Level D discount. The numbers in the second column shows the number of desktops in use by the organization.

EA level A 250 – 2.399
EA level B 2.400 – 5.999
EA level C 6.000 – 14.999
EA level D 15.000 +

Step-up licensing 
customers with a volume agreement including Software Assurance can upgrade the edition of software for a reduced price. For example they can upgrade from Windows Server Standard Edition to Datacenter Edition.
more info here 

Points
Some agreements like Select Plus use points to determine if a customer qualifies for the agreement. Each Microsoft product has points. These are documented in the Product List.

Pricelist

Many pricelists are available on Internet. Prices are hard to understand as you will see different prices for what looks like the same product. As you have learned the price depends on type of organization, agreement, enrollment, number of licenses in use etc.

This is a pricelist of Kernel Software

 

Dell starts partnering with Nutanix. Will use Nutanix NOS for Dell systems

At Dell User Forum 2014 Dell announced several new partnerships with Oracle, Fusion-io , Cloudera and the most interesting one : Nutanix.

Dell will OEM Nutanix Operating System (NOS) and install it on a line of new converged Dell systems. Dell plans to offer customers the Dell XC Series of Web-scale Converged Appliances, which combine compute, storage and networking into a single offering, powered by Nutanix software. The Dell XC Series of Web-scale Converged Appliances have planned worldwide availability during the fourth quarter of 2014.

Dell will be taking the calls for support to eliminate finger pointing. Dell is looking at integration of Nutanix NOS with Dell System Manager.

Nutanix has a blog titled “SDS for the masses: why Dell matters

Channel Register has a good story about this news here.

Web-Scale IT seems to be another new buzzword. It has nothing to do with hosting websites. I guess the name comes from the world wide web and companies like Google, Facebook and Amazon who made the growth of the web possible using low cost software defined architectures.

Gartner estimates by 2017 , 50 % of global enterprise will use this architectural approach.

What Web-scale IT means is a hyperconverged system (storage and compute in a single box) on x86 hardware with  software smart enough to recover from failures and able to scale out very easily with minor setup and maintenance. It has a rich API enabling automation and rich analysis. See the Nutanix PDF for more details.

The architecture comes from Google, Facebook, Amazon which uses cheap hardware and intelligent software to scale.

On Wednesday  June 25 Nutanix has a Web-scale Wednesday webinar event with many speakers. Registration is free and you even get a cool t-shirt!

Dell pressrelease here

Nutanix release here  and here

VMware vExpert 2014 Q3 applications are now open

VMware awards people for their contribution to the community by  the vExpert title. Becoming a vExpert can be done for example by writing blogs, do presentations, participate in a  VMUG or write a book.

Being a vExpert has numerous advantages. Besides being member of a select community and recognition, you will receive free Not for Resale licenses for various VMware and third party vendors, have access to VMware beta programms, be invited to the vExpert party at VMworld, have access to all content of Pluralsight including training of Trainsignal.

New is the possibility to apply each quarter for vExpert. Formerly people could apply only once per year. Applications for vExpert 2014 Q3 will be accepted up until September 13th at midnight PST.

More info here.

Azure Site Recovery is now available as Preview

Azure Site Recovery is now available as Preview. This service formerly known as Azure Hyper-V Recovery Manager is able to orchestrate failover between two customer owned sites.

It is also able to replicate Hyper-V virtual machine to Microsoft Azure datacenters saving customers on costs of a secondary datacenter.

Besides replication to Azure the service also allows an orchestrated recovery of virtual machines in case of failover to an Azure datacenter.

Customers must be using System Center Virtual Machine Manager 2012 R and virtual machines running on Hyper-V. At the moment only Generation 1 VMs are supported. VHD & VHDX virtual disk files are supported.

A getting started with Azure Site Recovery  for on-premises to Azure is available here.

More information on Site Recovery here.

Costs
During Preview customers ger a 50% discount on pricing. The costs of protecting on-premises virtual machines to Microsoft Azure are $ 27,- per month per virtual machine *during Preview* . Customers receive 100GB of replication and storage per VM. Charging is based on an average usage per month. Suppose a customer protects 20 virtual machines for the first half of the month and none for the second half, the average daily number of protected virtual machines being charged by Microsoft would be 10.

Costs for using Azure Site Recovery as an orchestration tool for replication to another customer managed site is $ 16,- per month per virtual machine.

Book
I will cover Azure Site Recovery in my new book on Microsoft hybrid cloud.

System Center, Hyper-V , Azure Meat(ing) June 26 the Netherlands

If you like System Center, Hyper-V, Azure as well as meat, don’t forget to register for System Center Summer Night “The MasterChef edition” which is rapidly approaching. This event is held in Culemborg, the Netherlands at June 26.

Do not worry about the Worldcup, the Netherlands is not playing. Below the games scheduled for June 26.

wc-soccer

At June 26 nine experts, seven of them are MVPs, will present five interesting presentations.

Because we have plenty of space left we invite those who have registered to bring a friend without additional cost. If you haven’t registered yet, please do and have a great afternoon which is completed with a nice barbecue.

 

Date of event

June 26th

 

Program

15:00 – 16:15 How many System Center fits on one grill by Ronny de Jong / James van den Berg [MVP] / Helmer Zandbergen / Marc van Eijk [MVP] / Dieter Wijckmans [MVP]

16:15 – 17:30 How Service Manager can do everything you need – a best-of-the-best Swiss cheese selection by Marcel Zehner [MVP]

17:30 – 17:45 Break

17:45 – 19:00 Light up the fire on your Hyper-V by Hans Vredevoort [MVP] & Peter Noorderijk on Hyper-V Architecture

19:00 – 20:15 Become a Masterchef on Microsoft Azure Automation by Maarten Goet [MVP]

20.15 BBQ time

This event is organized by SCUG.nl and Hyper-V.nu

 

Registration

Please register at https://www.eventbrite.nl/e/tickets-system-center-summer-night-2014-9265847399

 

Location

Unieplaza
Multatulilaan 12
4103 NM Culemborg

Google map here

Hyper-V 2012 R2 virtual machines lose randomly network connections . Be carefull with Emulex NICs! New driver expected in July

<update June 24>
Hyper-V Program Manager Ben Armstrong made a blogpost about this issue titled Hyper-V Network Connectivity Issues with Emulex Adapters

He is blogging about this for two reasons:

  1. I have been contacted by a number of customers who have hit this, and want people to know about what to do.

  2. It is really good to see a hardware vendor documenting status and workarounds for known issues, and I am glad to see this post going up.

<update June 20> 
Hans Vredevoort of Hyper-V.nu wrote a blog about the conference call he had with Emulex. Emulex was unaware of this issue for a long time. It seems HP did not inform Emulex about the issue. When Emulex was aware they could n0t reproduce the issue at first, and then found other issues as well.
His blog here: Additional Background on the VMQ Issue with Emulex and HP

<update June 19>

Emulex posted a blog on the Emulex website explaining the issue described in my post. The workaround is to disable VMQ. We knew that one for a couple of months.

The good news is that a new driver and firmware is expected to be released in Mid July.

See the blog here. 

————————

<update June 17>

I have been in touch with the CEO of Emulex about this issue. He stated “My team is very aware of this and while you may not have been provided the update you deserve, the issue has not been ignored.   I know the team has been very engaged with HP and MSFT on this ”

Lets hope there is some progress on resolving the issue

————————

Virtual machines running on Windows Server 2012 R2 Hyper-V could randomly lose their network connection. The only workaround to restore network connectivity is to perform a Live Migration of the affected VM to another host or to reboot the Hyper-V host.

To do this some ITpro’s wrote scripts which pings all VMs and if no response is received a Live Migation is performed.

In many cases the issue is seen on Emulex NIC’s in HP Gen8 blades on which Windows Server 2012  R2 with Hyper-V is installed.

The problem seems to be related to the number of VMQ’s available in the network interface. If the number of netadapters/ virtual nics in VM exceeds the number of VMQ’s available, some  virtual machines will lose network connectivity.

I found an explaination of how VMQ works here. For much more indepth details about VMQ see this blogpost and the earlier linked in the article.

 

Since the emulex netadapters have 16 VMQ slots total, the first 4 slots are taken up by the host OS. The first of the 4 is supposed to be “special” (i’ll get back to that in a bit). The other 3 are regular adapters. The next 12 are regular VM adapters. Each guest VM is assigned one VMQ slot out of 16.

4 + 12 = 16; all VMQ slots are assigned.

When the 13th VM tries to get a VMQ slot, it fails to receive one.

What’s supposed to happen, is the hypervisor is supposed to just start sharing it’s “first” slot (the special one), with any additional VM’s that can’t get VMQ slots (or any that have VMQ disabled).

What actually happens, on the emulex or broadcom adapters, is that the guest OS simply fails to allocate a VMQ slot, and fails to get any network connectivity at all. It can not talk to the host OS (even if it’s on the same VLAN and not communicating through the physical ports).

Basically, the Emulex and Broadcom give you exactly the VMQ slots avaialable and the “fail-over” technology of failing back to vRSS-like queues for the other VM’s simply fails to work, and any VM that wasn’t issued a direct VMQ fails to communicate.

The intel drivers correctly share the first VMQ slot with any additional VM’s. It ends up with higher-than-normal CPU usage on the first core, but that’s no different than how Windows Server 2008 R2 (or 2012 R2 with vRSS networking) works anyway.

I understand the  Emulex adapters currently support up to 30 VMQ on Windows Server 2012.

The workaround which works for many people is to disable VMQ on the nic by using this command

get-netadapter | disable-netadaptervmq

This blogpost by Ben Gelens describes the same issue. Ben solved it by disabling the  Virtual Machine Queue (VMQ) on just the management nic.

The issue is described at Hyper-V.nu and at aidanfinn.com . It is also reported on the Microsoft TechNet forum.

It seems to occur mostly when Emulex network interface cards are used. These are for example used in HP (HP 554M , HP 554FLB, 554FLC adapters use the Emulex chipset) , Broadcom NetXtreme 57xx NICs, and IBM servers. Especially the 10 GbE cards are suspect for this issue.

Emulex driver versions 10.0.430.570 ,  10.0.430.1003 and 10.0.430.1047 all seem to suffer from this issue. Some information on Emulex adapters in a Hyper-V environment using RSS and VMQ.

Also NICs of Broadcom and Intel are reported having this issue but likely less frequent.

It seems that virtual machines which handle a lot of network traffic are more affected by this issue than virtual machines which do not handle a lot of network traffic.

The probolem is experienced by many people.

There is at the moment no solution but waiting for Emulex to release a new driver.

Some other advises which I found on various sites and might or might not help. There are other network issues reported as well on various blogs. Some servers get a BSOD but this could possibly be resolved by using a Microsoft hotfix

  1. disable encapsulated packet task offload per Disable-NetAdapterEncapsulatedPacketTaskOffload cmdlet
  2. Disabled Large Send Offload v2
  3. Set-NetOffloadGlobalSetting -TaskOffload Disabled

 

Microsoft DirSync to be replaced by Azure Active Directory Sync Services

Microsoft is actively working on enhancements to connect on-premises Active Directory to Azure Active Directory.

DirSync and Active Directory Federation Services are two options to connect both. DirSync can now be used as a backup for ADFS. See my post here.

Microsoft is working on a replacement for DirSync. DirSync is a software tool used to synchronize objects located in an  on-premises, single forest Active Directory  to Azure Active Directory. Azure Active Directory is the Microsoft multi-tenent cloud version of Active Directory used for identity management for services like Office 365.

DirSync is basically an implementation of Forefront Identity Manage but with limited features. For example it is not able to sync objects of multiple on-premises AD forests nor is it able to handle multiple Exchange organizations.

To support these scenarios enterprises are at the moment required to use Forefront Identity Manager. However, configuring FIM can be challenging and can take considerable time.

The new tool which replaces DirSync will be named Azure Active Directory Sync Services or AADSync.  AADSync significantly simplifies the configuration and makes it more predictive.

Microsoft Azure Active Directory Sync Services (AADSync) is used to onboard an on-prem environment to Windows Azure Active Directory and Office 365 and continue to synchronize changes. It is used for more advanced scenarios where DirSync does not provide support, for example multiple on-prem AD forests. At the moment AADSync does not support multiple Azure subscriptions.

AADSync will also be able to synchronize Exchange Global Address Lists. Support for PowerShell is also available, it has about 58 commands.

Microsoft Azure Active Directory Sync Services is currently available in customer technology preview (CTP).  This is a first beta release.

You can join the Azure Active Directory Sync Services preview here. The AADSync preview will then be added to your Microsoft Connect account. Through this you will be able to download the most recent version, get information on known issues and updates, as well as provide feedback.

Currently AADSync is in beta. You may not use this release in a production environment unless agreed to by Microsoft. For customers participating in the TAP program, the beta can be used in production.
To be considered for the TAP program, please contact the feedback alias AADSyncFB@microsoft.com.

Mind AADSync does not have these features at the moment:

    • Exchange hybrid co-existence is not available.
    • Compared to DirSync, the following features are not available:
      • Password synchronization
      • Self-service password reset write-bac

More information on AADSync here.

Documentation on AADSync can be found here 

 

VMware releases vCenter Server 5.5 Update 1b

VMware released  vCenter Server 5.5 Update 1b at June 12. This update addresses Open SSL vulnerability and fixes some other issues.

The release notes are here
Download the bits here.

Resolved issues:

Security

Update to OpenSSL library addresses security issues
OpenSSL libraries have been updated to versions openssl-0.9.8za, openssl-1.0.0m, and openssl-1.0.1h to address CVE-2014-0224.
Server Configuration
Host profile answer file might not be applied when you reboot the ESXi host

After you restart the vCenter services, if you use Auto Deploy to reboot a stateless ESXi host, the host profile answer file might not be applied to the ESXi host and the host configuration might fail. This issue occurs if the reference host is not available.

This issue is resolved in this release.

Virtual SAN
Under certain conditions, Virtual SAN storage providers might not be created automatically after you enable Virtual SAN on a cluster

When you enable Virtual SAN on a cluster, Virtual SAN might fail to automatically configure and register storage providers for the hosts in the cluster, even after you perform a resynchronization operation.

This issue is resolved in this release. You can view the Virtual SAN storage providers after resynchronization. To resynchronize, click the synchronize icon in the Storage Providers tab.

What are the dangers of snapshots and how to avoid?

VMware vSphere snapshots can be very useful. A snapshot captures just like a photo  does the state of a virtual machine at a certain point in time. This capture cannot be modified while the virtual machine is active as it is read only. Returning to a state which is known to be good is a matter of a few mouseclicks.

However, snapshots are not that innocent. You can shoot yourself in the foot when not realizing the side effects of snapshots.

Backup software is a major culprit for causing issues with virtual machine performance and availability due to using snapshots. See my previous post about the impact of snapshots.

It is very important to understand what the impact of snapshots can be on availability and performance of virtual machines:

  • a virtual machine with active snapshot(s) performing many writes to disk can fill up capacity of a datastore causing all vm’s on a datastore to crash or pause
  • deleting a snapshot can pause a virtual machine for many minutes. This can for example result in Exchange Server DAG cluster failover or other unwanted side effects.

This post will provide information on snapshot deletions (commit as well as consolidation) and how to prevent pausing of virtual machines. We will focus on VMware snapshots but much in this post applies to snapshots of other solutions as well.

A couple of advises:

  • make sure if your application supports snapshots and under which conditions
  • a succesfull backup using snapshots does not automatically mean a succesfull restore!
  • snapshots are not a replacement for backups!
  • make sure snapshots are only active  for a couple of hours max. Then delete snapshots.
  • be very carefull using snapshots on virtual machines which perform many write transactions to disk
  • have a close look at impact and behaviour of your backup tool on snapshot files
  • make sure applications running in the virtual machine support snapshots. Snapshots of virtual machines running Microsoft Exchange are not supported. Snapshots of SQL server are supported only when VSS is used.
  • snapshots of virtual machines using in-guest iSCSI drives are not supported. 

 

Introduction to VMware snapshots

When a snapshot is made, the original VMDK (we call this the parent or base disk) is set to ReadOnly mode. All further writes to the virtual machine disks are stored in a delta disk (also called snapshot disk,  child disk, virtual  disk redologs or (sparse) delta disks). These delta disks have a <number> -delta.vmdk extension in the filename. Snapshots grow in chunks of 16 MB each. Each time a chunk is added the VMFS volume is locked.

Muliple snapshots can be taken of the same virtual machine.

Snaphots are very useful for making sure a known working situation can be restored. This because the parent disk does not change after the snapshot was taken (it is read only).

When a snapshot is deleted (we do not want to revert to the original situation when the snapshot was made), ESX(i) will merge the data written in the delta file back to the parent disk. A snapshot delete is also called a ‘commit‘ or ‘consolidation’.

While this is in progress, another delta disk is created which is used during the commit to store new writes. This is a ‘Consolidate Helper snapshot’  It is created at the moment a snapshot file is being commited to the parent disk. New incoming writes are stored in the consolidate helper snapshot file. Those are commited as well when the initial snapshot file has been succesfully commited.

 

 

 

 

 

 

 

To keep track of snapshot files ESX(i) uses a .vmsd file which is used for storing information and metadata about snapshots.

If an administrator wants to restore a certain state of a virtual machine (go back in time) , this is called revert.

This is a great article explaining what is happening under the hood of snapshots. This VMware KB article is also very informative.

Microsoft SQL and Exchange  support for snapshots 

Mind not all applications support snapshots. Microsoft policy on snapshots depends on the product. SQL Server supports snapshots which uses the VSS. This is the support policy for SQL Server.

SQL Server supports virtualization-aware backup solutions that use VSS (volume snapshots). For example, SQL Server supports Hyper-V backup. Virtual machine snapshots that do not use VSS volume snapshots are not supported by SQL Server. Any snapshot technology that does a behind-the-scenes save of a VM’s point-in-time memory, disk, and device state without interacting with applications on the guest using VSS may leave SQL Server in an inconsistent state.

Exchange Server (2010 and 2013) does not support snapshots. The quote below was taken from a Microsoft article . This is an article on Exchange 2013.

Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it’s running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots aren’t application aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange. As a result, making virtual machine snapshots of an Exchange guest virtual machine isn’t supported.

Especially when snapshots are taken of Exchange mailbox servers care should be taken. Snapshots of HUB and CAS roles should be okay in most cases.

If you make a snapshot of an Exchange Server and want to revert, there is a chance that after revert you will notice Exchange errors. If you are in bad luck Exchange might not be able to mount mailboxstores because of corruption.

When the snapshot is commited there is a chance the virtual machine has to be paused for a while. When an Exchange DAG role is installed in that virtual machine a DAG cluster failover might occur because the heartbeat is temporary lost.

If you want to make a snapshot of an Exchange server, make sure the virtual machine is shutdown first.

Snapshots of Microsoft Active Directory running on Windows Server 2012 are supported on certain versions of the hypervisor. See my post for more info. See this Microsoft post for additional info.

Out of sync situation

Snapshots are used by many backup solutions. However not all backup solutions clean up the delta disks after the backup of a vm has finished. Some tools just delete the metadata while the delta disks are still used to write data to.

VMware introduced in vSphere 5.0 ‘snapshot consolidation’. This corrects out-of-sync situations like a leftover snapshot file. Snapshot consolidation commits a chain of snapshot files to the original virtual machine parent file when Snapshot Manager shows that no snapshots exists but the delta files still remain on the datastore.

Snapshot consolidation is a very important task for administrators. Because snapshot files are still active these continue to expand and consume disk space untill the datastore runs out of space.

How do you know a virtual machine disk needs consolidation? It will be shown in the Summary tab of the vSphere Client.

vspehre-consolidation

A small explanation of how to use consolidation is shown in this VMware video.

Slow or paused virtual machines due to commit
Consolidation and snapshot commits could lead to a situation in which the virtual machine is paused for a few seconds or up to over 30 minutes!

This pausing is called a stun and is in certain circumstances required to be able to commit delta files.
Stunning is likely to happen when the guest operating system is performing more writes to the delta file than ESX(i) can commit to the parent disk. It is like a car driving max 50mph is trying to overtake a car driving an average of 60mph. To be overtaken the fastest car will need to stop or slow down for a while.

ESX(i) stun is a pause of the virtual machine so snapshots files can be commited to the parent disk. More info on stun in this VMware KB article.

VMware made several enhancements to snapshot commits in various releases of vSphere but still snapshots can have severe impact on virtual machines.

ESX(i) will try to commit snapshots without having to stun (pause) the virtual machine. Performing snapshot commits while the virtual machine is running is called asynchronous consolidate.

Initially the commit is performed during a period of 5 minutes. If this commit fails to get rid of all snapshot files because to many writes are coming in, it will do another try with a duration of 10 minutes. If this again fails because too much new writes are written, the snapshot commit duration is extended to 20 minutes. In total ESX(i) tries at a maximum of 10 times (called iterations).

Thereafter the virtual machine will be stuned. This is called a synchronous consolidate. Stunning means no new writes are coming in and ESX(i) is able to commit all snapshot files.

Beginning in ESXi 5.0, the snapshot stun times are logged. Each virtual machine’s log file (vmware.log) will contain messages similar to:

2013-03-23T17:40:02.544Z| vcpu-0| Checkpoint_Unstun: vm stopped for 403475568 us

In this example, the virtual machine was stunned for 403475568 microseconds (1 second = 1 million microseconds).

Avoiding stun or keep the stun duration as short as possible
If you do not want to stun / pause the virtual machine you can set  snapshot.maxIterations to 20 (or higher). This means vSphere will do more tries (iterations) to commit the snapshot files. More information in this KB article.

Be carefull to change settings and closely monitor the effects.

To do this:

  1. Shut down the virtual machine
  2. Right-click the virtual machine and click Edit Settings.
  3. Click the Options tab.
  4. Under Advanced, click General.
  5. Click Configuration Parameters and add snapshot.maxIterations

However, this could make things worse. Think again about that car (the commit process) trying to chase that other (leading) car (the writes from the os and applications in the guest). If the speed of the leading car remains higher than the chasing car, the longer the duration of the chase, the bigger the distance.

Alternatively you can set snapshot.maxConsolidateTime to 60 seconds. This means you can accept a pause of the virtual machine for 60 seconds to do a synchronous consolidate. This is often a better option than wait for the snapshot file grow so big the virtual machine will require to be stunned for a much longer time.

ESXi 4.1 has a update which added parameter snapshot.asyncConsolidate.forceSync = “FALSE” which needs to be added to the VMX file. This setting disables synchronous consolidate and the virtual machine will never be stunned. More info in this KB.

 

Some additional info
VMware published a remarkable number of knowledgebase articles on snapshots. Below just some examples.

VMware KB A snapshot removal can stop a virtual machine for long time (1002836)
VMware KB Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation (2010953)
V
Mware KB Delete all Snapshots and Consolidate Snapshots feature FAQ (1023657)
V
Mware KB Commands to monitor snapshot deletion in ESX 2.5/3.x/4.x and ESXi 3.x/4.x/5.x (1007566)
V
Mware KB Consolidating snapshots in vSphere 5.x (2003638)
V
Mware KB Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots (1018029)

Follow

Get every new post delivered to your Inbox.

Join 102 other followers

%d bloggers like this: