VMworld 2010: GD05 Group Discussion on Storage with Chad Sakac

VMworld  also  has Group Discussions. Less known than Breakout Sessions and certainly less crowded. Group discussions are interactive sessions lead by a Knowledge Expert. Group Discussion GD05 was held at Tuesday October 12 and presented by Chad Sakac of EMC. The whole group discussion has been filmed at uploaded to YouTube by user DuneStudio45. Thanks for sharing! 

Part 1 of 5: 15 minutes, 6 seconds http://www.youtube.com/watch?v=Rn9Ygs1uGVY
Part 2 of 5: 15 minutes, 3 seconds http://www.youtube.com/watch?v=eCPEBgcHZhs
Part 3 of 5: 12 minutes, 59 seconds http://www.youtube.com/user/DuneStudio45#p/a/u/2/y-MgDkf4r9k
PArt 4 of 5: 15 minutes, 2 seconds http://www.youtube.com/user/DuneStudio45#p/a/u/1/Ur8KOeOTmJA
Part 5 of 5 : 12 minutes 13 seconds http://www.youtube.com/watch?v=i6sHQz2aQas

VMworld 2010: TA8133 Best practices to increase availability and throughput for VMware

This session was very popular. The hall was completly full with I guess 600 seats occupied. Co-presented by Chad Sakac of EMC and Vaughn Stewart of NetApp. Chad has one of the best if not the best website on storage called Virtual Geek http://virtualgeek.typepad.com/. Vaughn has a website called the Virtual Storage Guy at http://blogs.netapp.com/virtualstorageguy/ Both sites deserve to be bookmarked for ever or to be added to your RSS-feeds.

The presentation was quite funny. Chad and Vaughn call themselves frenemies, friends and enemies at the same time. It was clear they are good friends even working for competing companies. They explained FiberChannel, iSCSI and NFS are all three good solutions. Just pick the one you are most comfortable with or which suits your budget and demands best.

I guess the most important lesson in this presentation was aligment. It has been said and written about many times. Make sure your datastores are aligned with the underlying storage and that the virtual machine disk files are aligned with the  datastore. If a VMFS datastore is created using vSphere Client it is aligned automatically. Windows Server 2008 aligns automatically as well, older operating systems do not.

If you know thst virtualization will cost around 10 to 30% of performance compared to physical servers a lot of people will reconsider virtualization. But for storage they seems to less care. In fact not aligning will cost in worse case 10 to 30 % of storage performance. Using tools like diskpart aligment can be done.
When performing physical to virtual (P2V) conversions make sure the tool you are using aligns properly. Most free tools do not align. PlateSpin Migrate for example aligns the virtuals disk of the vitual machines. More on alignment here http://vmtoday.com/2010/06/storage-basics-part-vii-storage-alignment/ and here https://up2v.wordpress.com/2010/05/05/platespin-migrate-best-practises/

Chad and Vaughn talked about MPIO and settings like round robin and MRU. Also Asymmetric logical unit access (ALUA) was mentioned. Make sure you know what this does!
Interesting enough and opposite I always believed was that the use of Jumbo Frames does not matter much in performance benefit. Chad and Vaughn advised not to use it. For Jumbo Frames to be effective you will need the network adapter, switch and storage array.

As a frequent reader of the blogs of Chad and Vaughn I did not learn much new during this session but I think a lot of others did.

The PowerPoint presentation can be downloaded here http://www.google.nl/url?sa=t&source=web&cd=7&ved=0CDoQFjAG&url=ftp%3A%2F%2Fftp.documentum.com%2Fvmwarechampion%2FEvents%2FVMworld%2FVMworld_2010%2FAmericas%2FTA8133_Sakac_Stewart%2Ffinal%2520content%2FTA8133_Sakac_Stewart_Final.pptx&ei=x1rETKDcGYeWOtjn7PQL&usg=AFQjCNE4RgEMKmNhhdt2Iyp87Z-3d3NkMw

VMworld 2010: TA8233 Prioritizing Storage Resource Allocation in ESX-based Virtual Environments Using Storage I/O Control

This was one of the best sessions at VMworld 2010. Two realy good presenters and an interesting topic. Storage I/O Control is a new feature of ESX4.1 available only in the Enterprise Plus Edition of vSphere. What this feature does is control disk access to datastores. SIOC is limited to datastores accessible using iSCSI and FC, so blockbased storage. NFS being filebased storage is not yet supported. However, in the VMware labs SIOC control for NFS is already running so expect NFS support to be added soon. 

Why is Storage I/O control such an important feature? Because 90% of all performance issues in which VMware is involved is related to storage! Storage can be seen as the bottle neck of virtualization.

Performance problems related to storage can be seen as noisy neighbors. A single virtual machine can dominate the usage of available storage resource in such way other virtual machines have insufficient i/o access. Those VM’s can be running business critical applications while the VM dominating the usage of storage i/o has an un-important role. You like to control this noisy neighbor so he or she becomes much more quiet!

Storage I/O control is difficult to develop. It took VMware 3.5 years to develop this feature. And it is very simple to enable. SIOC  is disabled by default. It can be enabled by selecting a datastore and enable SIOC.  The recommended setting is to enable it.

As soon as SIOC is enabled, access to storage is equally divived over the virtual machines sharing the same datastore which is SIOC enabled. So a VM dominating I/O will get less  I/O than before SIOC was enabled. To control priorization you can use shares or set the number of IOPS a virtual machine can maximum get.

SIOC starts becoming active (kicks in) when the latency is 30ms or more. VMware found out that 30ms is on average the limit for acceptable performance. Latency more than 30ms results in bad performance. It does not make sense to activate SIOC earlier. It is like a VIP lane on a highway. When there is no congestion, so traffic runs smoothly, the usage of a VIP lane or carpool lane does not have any effect. The same counts for SIOC.

Using shares the access to storage can be priortized. Access is not a static thing. If a virtual machine which a high number of shares, so with a relative high priority is actualy not consuming I/O, other VM’s having a lower share, will get those I/O’s. So SIOC is dynamically assigning I/O based on actual consumption. To control this the host device queue depth is adjusted.

The presenters show different slides showing the effect of not using SIOC and using SIOC. Even when SIOC is enabled, access of I/O for vm’s having a high number of shares will drop. The lose  of I/O access can be controlled by adjusting the latency value. By lowering the latency the moment SIOC kicks in can be controlled.

A lot more information on SIOC can be found on the excellent Yellow-Bricks.com

Storage I/O Control best practices http://www.yellow-bricks.com/2010/10/19/storage-io-control-best-practices/
Storage I/O Fairness http://www.yellow-bricks.com/2010/09/29/storage-io-fairness/
SIOC, tying up some loose end http://www.yellow-bricks.com/2010/10/08/sioc-tying-up-some-loose-ends/ 

Info on vkernel.com
What to Expect When You Enable Storage I/O Controls in ESX 4.1  http://blog.vkernel.com/2010/09/what-to-expect-when-you-enable-storage.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+vkernel/FnyZ+(VKernel+Virtualization+Blog)

VMworld 2010: BC8675 SRM Futures: Failback Automation, Workload Mobility and More

This session was presented by VMware’s Lee Dilworth and Jacob Jensen. Site Recovery Manager (SRM) is a VMware solution which enables automated recovery of a datacenter running VMware virtual machines. Recovery plans can be tested without disturbing the production environment. SRM works together with storage vendors to enable replication of virtual machine diskfiles at the storage level.

A new version of SRM is currently in beta. If someone is interested in participating in the beta Lee can be contacted.

Lee explains that in a future version of SRM reconfiguration of IP-addresses is not needed anymore. Currently SRM customers have two options. Either use a stretched VLAN which enables the use of the same IP-subnet in both protected and recovery site. The alternative is to re-IP the virtual machines in case they are running in the recovery site after a failover. This re-IP can be done automatically by SRM but beforehand an Excel sheet needs to be filled in with the new IP-configuration. A recent customer survey showed that 60% of the SRM customers are using a flat network and 40% do Re-IP.

As said, in the near future when Cisco techniques like Virtual Private LAN Services (VPLS) or Overlay Transport Virtualization (OTV) are used Re-IP is not needed anymore.

SRM will support failover to cloud services. Currently a SRM customer will need two sites; one primary datacenter (protected site) and an alternate datacenter for recovery. Small and medium businesses will not always have a second datacenter. In the future SRM will be able to replicate virtual machines to a cloud provider. In case the datacenter is not available virtual machines can be made operational in the datacenter of a cloud provider.

Also integration with vCloud Director is planned to be able to protect virtual datacenters created with vCloud Director.

Fallback is something on the wishlist for a lot of customers. Currently fallback is possible but it will take some manual steps. In the future fallback will be automated.

The next version of SRM will have 5 groups for recovery priority. Each group can be assigned a different priority for recovery. Currently there a three levels of priority (high, normal, low). It will also be possible to indicate dependencies of start up of virtual machine inside groups. So if in Group 1 three virtual machines are placed, it will be possible to have a SQL server start only after a Active Directory server has started.

Host based replication is one of the most exciting new features of the new version of SRM. Instead of needing a storage array which is able to replicate, using host based replication single virtual machines running on an ESX host local VMFS datastore can be replicated to another site. This is especially usefull for branch offices where a few ESX hosts are placed using local storage or a storage array not capable for storage replication. Host based replication enables a Recovery Time Objective of 15 minutes to 24 hours. For the initial replication it will be possible to create an OVF file of the virtual machine, ship it to the datacenter and only replicate the changes. This will reduce the amount of data to be replicated over sometimes low bandwidth connections between remote office and datacenter.

The user interface has been adjusted. Recovery plans for example can be seen at both recovery and protected site.

Performance of SRM has been increased significantly. A lot of code has been rewritten and that results in a much better user expericence. The next version of SRM has been so re-enginered it is ready for the future!

VMworld 2010: SP9741 The new standard for data protection

<update October 20> For a very good and comprehensive review of version 5 of Veeam Backup & Recovery see this link https://up2v.wordpress.com/2010/10/20/cool-review-on-veeam-backup-replication-v5/

This session at VMworld 2010 was presented by Veeam’s Doug Hazelman. Doug explains the audience about the new Veeam product called Veeam Backup & Replication v5. This new release has several new and interesting features which are unique for the industry and a revolution for protection of virtual machines running on VMware vSphere.

Version 5 has three new technologies:

SureBackup Recovery Verification. Why does someone makes backups? To be able to restore the data. How are you sure you can restore your data? By actually restoring the data. How many times does someone actually test a restore procedure? I do not think many people do that. A lot of companies find out their backup is not useable at the moment they really need to recover the data. Finding out the hard way is not good.
SureBackup will actually restore a virtual machine after every backup, start the virtual machine, start services and report if data and services are available. The product does this by:
– storing the backup of virtual machines on a NFS volume managed by the Veeam backup server (prefered to be a physical server)
– attach the NFS volume as a VMFS datastore to ESX hosts.
-compress the virtual machine data
-start the virual machine from the backup datastore while being compressed and in an isolated network so it does not conflict with the server it made a backup of. The virtual machine virtual disks on the Veeam NFS datastore will remain read only to guarantee the integrity of the backup.
Instant VM recovery restores an entire vm in a couple of minutes.  

U-Air is a Universal Application Item recovery technique enabling the recovery of individual items from any virtualized application without the need for backup agents in the VM. Veeam has a wizard available which enables item recovery of :
-Microsoft Active Directory
-Microsoft SQL Server
-Microsoft Exchange Server

For other applications, the procedure for item recovery will be to restore the virtual machine holding the application in an isolated network and use tooling to extract the data by the application manager. Databases like SQL Server have tools which can easily export tables.
In the near future more wizards will be added.

I have seen a demo of the product and it is really cool. Very easy to use.

Veeam Backup and Replication is not able to backup physical servers, is limited to backup of VMware virtual machines. It is not able to handle tapes and tape libraries. You will need an additional product for tape handling if required. It needs to store it’s backup data to disk and supports  destinations like Windows shares,VMFS datastores located on iSCSI or FC or NFS.

Doug Hazelman announced for the first time that version 5 of Backup & Replication will be released at October 20. To join the lauch event, click here http://www.veeam.com/go/v5-launch-event/

More on the launch event here http://www.veeam.com/blog/we-are-excited-the-v5-live-launch.html

VMworld 2010: TA8661 Deploying vSphere in a ROBO Environment

This posting will give a global overview of the presentation called TA8661 Deploying vSphere in a ROBO Environment given at VMworld 2010 in Copenhagen.

The presentation was given by two VMware employees from the UK; Paul Nothard and Rob Upham. It discussed the options and challenges of deploying vSphere in a remote office or branch office. A couple of customers of Paul and Rob have small offices, like shops or postoffices. For deployment of IT several options are available:

  1. install ESX on a couple of hosts and have vCenter Server manage hosts and virtual machine from within the branch office
  2. install ESX on a couple of hosts and centally manage hosts and VM’s using vCenter Server running in a datacenter
  3. centralized computing in which no servers are located in the branch office.

For management option 3 is the best option. All data and applications are securely running in the datacenter. However not all application are ready to run in a datacenter over sometimes small bandwidth connections. Some offices are actually retail shops with Electronic Point of Sales (EPOS) systems.

To have vCenter Server running in all remote offices would be expensive because of licensing and also not ideal for management.

Option 2 would be the best, and option 3 should be the target for the future.

Several challenges exist when vCenter Server is used from a central location. Deploying the vCenter Server agent to the ESX hosts will consume some bandwidth for a limited time.  Some retail shops are so limited in available bandwidth that deploying the agent during opening hours could lead to problems with for example the EPOS system. So be carefull with that.

For regular management there are hardly any issues using a central vCenter Server instance over relative low bandwidth connections (128 Kbps). Deployment of updates and patches using vCenter Update Manager can be a challenge as updates will need to be downloaded to every ESX host. Using a PowerShell script multiple downloads of the same patches can be avoided. See the script at Rob Upham site http://wirey.com/2010/03/patching-esxi-using-powercli/

The vSphere Essentials for Retail and Branch Offices is probably VMware’s best kept secret. Available since January 2010 this edition is a cost effective solution for rolling out vSphere in branch offices. It enables the license to install 3 hosts with 2 CPU’s each on 10 different branch offices. The ESX hosts can be managed using a central vCenter Server standard instance.

Read more about the Essentials for Retail and Branch Offices here https://up2v.wordpress.com/2010/10/18/managing-vsphere-4-1-cost-effective-in-remote-offices-and-branch-offices-robo/

Managing vSphere 4.1 cost effective in remote offices and branch offices (ROBO)

This article will give information on deploying vSphere 4.1 cost effective to remote offices and branch offices needing a max of 3 hosts.

If a company is having several branch offices /remote offices which are small enough to have a maximum of three hosts and 6 CPU’s and the need to manage those offices from a single vCenter Server running in the central datacenter, several editions of vSphere are available. Most people are aware of the 4 editions of vSphere 4.1 :
Standard, Advanced, Enterprise and Enterprise Plus editions which are targeted at Enterprises.

vSphere Essentials and Essentials Plus Editions can only be managed using the vCenter Server for Essentials and not from a central vCenter Server Standard!

For a small office a vSphere Standard Edition will deliver sufficient features. vSphere 4.1 has vMotion, HA and VMware Data Recovery. However, there are two less know editions available which are a cost effective alternative to the Standard Edition.
Released in January 2010 are two vSphere editions named:
vSphere Essentials for Retail and Branch Offices
vSphere Essentials Plus for Retail and Branch Offices

These software bundles comes with features which are equal to Essentials and Essentials Plus.  Essentials Plus for Retail and Branch Offices delivers HA, vMotion and Data Recovery. vSphere hosts licensed by the Essential for Retail and Branch Offices can be managed using vCenter Server Standard edition instance which is running in a central location, for example the enterprise datacenter. The Essentials and Essentials Plus editions are limited to be being managed only from the vCenter Server for Essentials (limited to management of up to 3 hosts)

There is a quite a difference in pricing. For a remote office with a max. of three hosts and 6 CPU’s the vSphere Essentials Plus for Retail and Branch Offices  Starter Kit including 1 year of basic support costs $ 12040,-. This enables the buyer to install 3 hosts with max 6 CPU in 10 different Retail Offices. At total of 60 CPU’s. The offices need to be in a different building. For each additional office an Add-on needs to be purchased which costs $ 1204,-

Per CPU this costs $12040 / (10 x 6) = $200,-

Compare this to the per CPU price of vSphere Standard Edition which is $ 1268,- including 1 year Basic suport.

For Essentials for Retail and Branch offices support and subscription (Sns) for minimal 1 year  is required. Initial purchase of 10 packages is also required. One package consist of the the right to install three hosts having 2 CPU’s .  Limit 1 package per physical building – No splitting licenses from 1 package across multiple buildings.

As said, to centrally manage ESX hosts placed in remote offices, Essentials and Essentials Plus is not an alternative. Those cannot be managed from another vCenter Server instance than the once that was purchased with the Essentials or Essentials Plus package. See the VMware ESX/ESXi EULA:

c) VMware vSphere Essentials and VMware vSphere Essentials Plus

If you have licensed VMware vSphere Essentials or VMware vSphere Essentials Plus, the following additional license terms apply:

VMware licenses VMware vSphere Essentials or VMware vSphere Essentials Plus editions (collectively “Editions”) to you solely for managing up to three (3) server hosts and your use of these Editions is limited to servers with up to two processors. For these Editions, the server hosts must be managed by the VMware vCenter Server that is provided with these Editions, and that same VMware vCenter Server cannot be used to manage other server hosts not included with these Editions.

An interesting whitepaper called “Simplifying Remote and Branch Office Management with VMware” can be found here:
It has several scenario’s for deployment of vSphere in branch offices.


solutions Brief

VMware Solutions for Remote and Branch Offices

Application Virtualization White Paper

VMware vSphere Essentials Data Sheet

VMware vSphere Product Brochure

VMware vSphere Pricing & Packaging White Paper
Managing Remote ESX Hosts over WAN with VMware vCenter

Update Manager host tasks might fail in slow networks

Virtualize Branch Offices with VMware and Riverbed|

Compliance 101: Considerations for Virtualized Environments
ROBO ESX Implementation in Kroger Stores (TA2222)
Leveraging VMware View and Cisco WAAS in a ROBO
deployment (DV1460)
Beyond the Datacenter: Virtualizing Your Remote and
Distributed IT Workloads (TA4020)
Architecting Remote and Branch Offices for HA and DR
New Trends in Remote Office / Branch Office with VMware
Remote Office Data Protection and De-Duplication for
VMware (TA3089)
Virtualization of Remote Sites (IP29)

VMworld day 2 report

Day 2 at VMworld did not start with a general session as it did last year. It had been cancelled at late notice. I cannot say this is a big miss. Last year the general session was a short of commercial in which several VMworld sponsors entered the stage to tell the audience how great their solution was. So this year it all started at 08:00 when the hand on labs opened and at 09:00 with breakout sessions.
I attended around 6 sessions. Some of them interesting, some pretty boring. I will make a posting on some of the interesting sessions like the one about Storage I/O Control. The one on Site Recovery Manager showed some new features.  Host Based Replication and the ability to fall back are the most appealing ones. Later more on that in a separate posting.

The second day ends traditionally with a big party. This time the party was given at the Forum, a big hall in the centre of Copenhagen. The theme was the sixties and seventies. Good food, drinks and entertainment.

VMworld day 1 report

Day one started as usual with the general session. This time the cloud was the buzzword at the keynote. I am not going in details as they can be found everywhere. The keynote video can be seen here http://www.vmworld.com/community/conferences/europe2010/generalsessions.

I did not hear anything new, except maybe there will be a vCenter client for iPad available soon.

This year more than 6000 persons have registered, a new record and much more compared to the 4700 of last year when the event was held in Cannes. I guess the event will not return to Cannes anymore as the venue  is just too small to handle so many people. A pity as I liked Cannes. Very comfortable cinema type of seats, nice conference rooms. In Cophenhagen lots of big halls, lots of concrete floors, plastic seats and large crowds. But the Solutions Exchange hall is nicely integrated into the space where the sessions are held and where the hands on labs are held.

The Solutions Exchange seems to be a bit smaller than last year in Cannes. Also I really miss some nice goodies. The usual t-shirts and pens are there, but I believe the more interesting goodies are not here this year. Never mind 😉 The reception party at the end of day one was quite nice. Lots of beer, lots of young ladies serving sate, meatballs, pineapple covered in chocolade.

Attended several presentations today. Hardly heard anything really new that is not available on the internet. Nowadays lots and lots of good information can be found on the internet. I noticed some information was repeated in several sessions.Storage I/O control is a nice new feature which will help to control the I/O needs of your virtual machines.

The best thing of this years VMworld is the hands on labs. Very well organized. Last year you had to report which lab you liked to do and had to wait quite a while before you could do the lab, depending on the popularity of the lab. Now all seats are able to deliver any of the labs. The whole organization seems to be very efficient. The quality of the labs varies. Some you are extremely focussused on perforing the tasks in the lab without being able to learn what the product actually does. Found the Orchestrator lab a bit complicated with losts of screen to be filled in. Will do some more labs tomorrow as labs cannot be done when VMworld finishes, while the presentations are available for viewing in a few weeks times.

%d bloggers like this: