VirtualPro

In today’s economic climate it’s currently the done thing to sweat existing assets for as long as you possibly can.  At the moment I am working on a vSphere deployment and we are recycling some of our existing ESX 3.5 U4 hosts as part of the project.  So over the weekend I was testing out vMotion between a new host with the Intel Xeon X7460 processor and an old host with the Xeon 7350 processor.  I was getting the following error message displayed which pointed to a feature mismatch relating to the SSE4.1 instruction set.  Thankfully the error pointed me to VMware KB article 1993

Within this KB article it immediately refers you to using Enhanced vMotion Compatibility (EVC) to overcome CPU compatibility issues.  I had never used EVC in anger and wanted to read up on it a bit more before making any further changes.  A quick read of page 190 on the vSphere basic configuration guide gives a very good brief overview for those new to EVC. 

So I was referred to VMware KB article 1003212 which is the main reference for EVC processor support.  Quite quickly I was able to see that EVC was supported for the Intel Xeon 7350 and 7460 using the Intel® Xeon® Core™2 EVC baseline.  In essence as far as vMotion is concerned all processors in the cluster would be equal to an Intel® Xeon® Core™2 (Merom) processor and it’s feature set.  This basically masks the SSE4.1 instruction set on the Intel Xeon 7460 that was causing me the problem.

So I set about enabling my current cluster for EVC,  however when I went to apply the appropriate baseline I was getting the following error displayed. The error related to the host that was currently running 3 Windows 2008 R2 x64 servers.  These servers were obviously using using the advanced features of the Intel Xeon 7460 and as such that host could not be activated for EVC.

The vSphere basic configuration guide (Page 190) makes the following recommendation for rectifying this issue, the example matched my situation exactly.

All virtual machines in the cluster that are running on hosts with a feature set greater than the EVC mode you intend to enable must be powered off or migrated out of the cluster before EVC is enabled. (For example, consider a cluster containing an Intel Xeon Core 2 host and an Intel Xeon 45nm Core 2 host, on which you intend to enable the Intel Xeon Core 2 baseline. The virtual machines on the Intel Xeon Core 2 host can remain powered on, but the virtual machines on the Intel Xeon 45nm Core 2 host must be powered off or migrated out of the cluster.)

Now here is the catch 22, my new vCenter server is virtual and sits on the ESX host giving me the EVC error message.  I had to power it off to configure EVC but I can’t configure the EVC setting on the cluster without vCenter,  how was I going to get round this?  Luckily VMware have another KB Article dealing with exactly this situation.  The aptly titled “Enabling EVC on a cluster when vCenter is running in a virtual machine” was exactly what I was looking for. Although it involved creating a whole new HA / DRS cluster complete with new resource groups, etc it was a lot cheaper than buying a large number of expensive Intel processors. It worked perfectly, rectifying my issue and allowing me to use all servers as intended.

Moral of the story…..… Check out VMware KB article 1003212 for processor compatibility before buying servers and always configure your EVC settings on the cluster before adding any hosts to the cluster.  If it’s to late and you have VMs created already,  well just follow the steps above and you should be fine.

VMware, vCenter, vSphere Cluster, CPU, ESX 4, EVC, Intel, vCenter, vMotion, VSphere

I’m currently working with my colleagues on an upgrade of our VI 3.5 infrastructure to vSphere Enterprise Plus.  We have recently been mulling over some of the design elements we will have to consider and one of the ones that came up was virtual Distributed Switches (vDS).  We like the look of it,  it saves us having to configure multiple hosts with standard vSwitches and it also has some nice benefits such as enhanced network vMotion support, inbound and outbound traffic shaping and Private VLANs.

One of the questions that struck me was,  what happens if your vCenter server fails? what happens to your networking configuration? Surely your vCenter server couldn’t be a single point of failure for your virtual networking, could it?

Well I did a bit of digging about, chatted to a few people on twitter and the answer is no it would not result in a loss of virtual networking.  In vSphere vDS the switch is split into two distinct elements,  the control plane and the data plane. Previously both elements were host based and configured as such through connection to the host, either directly using the VI client or through vCenter. In vSphere because the control plane and data plane have been separated,  the control plane is now managed using vCenter only and the data plane remains host based.  Hence when your vCenter server fails the data plane is still active as it’s host based where as the control plane is unavailable as it’s vCenter based.

One thing I was not aware of was where all this vDS information is stored . Mike Laverick over at RTFM informed me that the central config for a vDS is stored on shared VMFS within a folder called the .dvsData folder. I’ve since learnt that this location is chosen automatically by vCenter and you can use the net-dvs command to determine that location. It will generally be on shared storage that all ESX hosts participating in the vDS have access to.  As a back up to this .dvsData folder a local database copy is located in /etc/vmware/dvsData.db which I imagine only comes into play if your vCenter server goes down or if your ESX host loses connectivity to the shared VMFS with the .dvsData folder.  You can read more about this over at RTFM

Interesting links if your considering VMware Distributed Switches

VMware’s demo video of vDS in action, for those who want to learn more about vDS

Mike Laverick’s great reasoning on whether you should use vDS or not

Eric Sloof’s vDS caveats and best practices article

VMware’s vSphere Networking white paper explaining new vDS features

VMware’s vSphere Networking white paper on vDS network migrations

Jason Boche’s very interesting article on a specific vCenter + vDS issue

ESX, vCenter vCenter, VSphere

For those of you that will have heard of Alan Renouf you will undoubtedly know of his talents in the dark art of VMware CLI / Powershell.  For those of you who don’t know him I suggest you check out his web site  to sample some of the many great articles and scripts he’s already produced.

His latest powershell creation has recieved a lot of attention in the last couple of days and with good reason. The Daily Report is a configurable script where you can set thresholds and variables such as snapshot age, datastore space free thresholds or number of days to look at for vCenter warnings and errors.  The script when run goes off and examines your Virtual  Infrastructure based on these variables and then proceeds to email you a nice html report on the following items.

·         VMs created in the  x number of days and who created them.

·         VMs deleted in the  x number of days and who deleted them.

·         Datastores which have less than x% of free space remaining.

·         VMs that have CD-Rom or Floppy drives connected.

·         VMs with no VMware Tools installed.

·         Snapshots that are older than x number of days.

·         Current state of vCenter Services.

·         vCenter events that have been logged in x number of days.

·         Windows events  on the vCenter server that relate to VMware.

·         Hosts in maintenance mode or a disconnected state.

Get yourself over  to Alan’s site and download a copy of the script and give it a try,  I did today and the results were enough for me to go ahead and implement this as a scheduled task.  If you’d like to see more features in Alan’s Daily Report script then give him some feedback,  there are a few good suggestions on the blog post already and I’m sure the next version isn’t far away.  Great work Alan, keep it up!

ESX, VI Toolkit / Powershell, VMware, vCenter esx, vCenter, VI Toolkit / Powershell

Duncan Epping over at Yellow bricks has posted a most interesting article which I read tonight when reviewing my RSS feeds.  It instantly struck a cord with me because recently we have been having issues with an ESX host at a site remote from our recently upgraded vCenter 2.5 U3 server.

I just received an email from a fellow consultant about a customer which had vCenter tasks time-out every once in a while. At times also ESX hosts got disconnected for no apparent reason at all. He discovered the following article by Richard Blythe aka VMware Wolf: ESX disconnects randomly or when doing VI client tasks from VC, task randomly timeout after a long idle time. Richard created a list of issues/errors that might be related to this issue:

• ESX disconnects randomly from VirtualCenter
• ESX disconnects when performing VI Client tasks from VirtualCenter.
• Tasks randomly timeout after a long idle time
• “An error occurred communicating to the remote host” pops up.

The article refers to an issue with vCenter Update 3 in combination with firewalls using state-ful inspection. The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX. The official KB article hasn’t been released yet but a temporary workaround has been published by Richard. If you run into any of the before mentioned issues head over to Richard’s website and try out the workaround until the fix or official KB article is released.

When conducting operations with this particular host using VI client attached to vCenter server,  we get “An error occurred communicating to the remote host” pop up more often than not.  I have been looking through the logs in vCenter for this host and it appears as well as manual tasks, our overnight Platespin protection replication jobs are also getting this message when executing.  This might explain some of the issues we’ve been having with some of our newer replication jobs not completing.

I’ve had a quick look at VmwareWolf’s workaround and have asked Richard if you need to create a dummy vm on each host or just the hosts that experience the problem as its not completely clear.  If i get a response I’ll let you all know what it is, meantime we look forward to an official KB hot fix release from VMware

* UPDATE - 09/02/09*
Richard at VMwareWolf came back to me and informed me that the dummy vm only needs to be setup on the affected host.  I set ithe workaround up yesterday and it appears to have resolved our issue for a job that was consistently reporting the error.  A permanent fix is still outstanding from VMware.

VMware, vCenter vCenter

I was looking into an issue following an upgrade to vCenter Server 2.5 last weekend.  So I set about searching through the file system for the log files on the server with very little luck to be honest.

I then found two excellent posts from Rick Blythe a.k.a the vmwarewolf,  the posts detail the locations of the logs and what each one means.  This is an excellent post and one that I’m going to keep handy for all those strange little issues where insight into the logs might give a clue to the problem.

Virtual Center Logs
http://www.vmwarewolf.com/which-virtual-center-log-file/

ESX Server Logs
http://www.vmwarewolf.com/which-esx-log-file/

ESX, ESXi, VMware, vCenter esx, vCenter, virtual center

Well this weekend I had the job of updating our Virtual Center deployment from 2.0.2 to 2.5 update 3. The primary reason for this was to prepare for the introduction of ESX 3.5 hosts into the Virtual infrastructure.

Due to the importance of our Virtual infrastructure I decided to do as much reading and preperation as possible to ensure it all went smoothly. It’s easier to convince our change management team to let us make changes if we get them right first time,  so this one was important to facilitate an easier path for future change.

So what did I read to ensure I’d covered everything,  here are a few links to get you started

Mike Laverick’s Upgrade Experience PDF
http://www.rtfm-ed.co.uk/?p=482

vCenter Server 2.5 Update 3 - Release Notes
http://www.vmware.com/support/vi3/doc/vi3_vc25u3_rel_notes.html

ESX 3.5 and vCenter Server 2.5 Upgrade Guide
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_upgrade_guide.pdf

The above PDF is a brilliant guide to the process you should follow, including rollback.  You should read this thoroughly so you understand all the pre-requisites and can avoid those silly problems that could cause your upgrade to fail.

Of course vCenter Server 2.5 introduces Update Manager. Although we can’t use it as we have ESX 3.0.1 and 3.0.2 hosts ( supports 3.0.3, 3.5 and 3.5i only ) I decided to install it anyway so it’s there for the future.

Here are some of the links I used to plan out my update manager deployment.

Update Manager Administration Guide
http://www.vmware.com/pdf/vi3_vum_10_admin_guide.pdf

Update Manager Performance and Best Practice Guide
http://www.vmware.com/pdf/vum_1.0_performance.pdf

Update Manager Size Estimator
http://www.vmware.com/support/vi3/doc/vi3_vum_10_sizing_estimator.xls

One of the main mistakes that people tend to make is to not give the SQL accounts the correct permissions on the new update manager database and the MSDB database.  Make sure you cover this one or you upgrade will fail.

It all went quite smoothly, I initially had a couple of issues which appeared to be related to me attempting to do a custom install. I wanted to ensure I could go through all settings and customise as required, the install however failed with various MSI error messages. I started the install again and didn’t choose the custom setup this time. This however created an issue whereby the Update Manager database appeared to install as SQL Server 2005 Express.  I wanted to put it on the same SQL 2000 server as our Virtual Center database but I never got the option as far as I can remember.  I have today uninstalled Update Manager and re-installed it using the “use an existing database” option and the SQL 2000 database.  It worked fine the second time around.

No immediate problems following the upgrade,  I had read some horror stories about issues with the Virtual Centre Agent on the host not updating.  Luckily for me it wasn’t an issue. Good luck with your upgrade

VMware, vCenter vCenter, virtual center, VMware