Archive for the ‘ESX’ Category

EMC PowerPath/VE – MpxEsxPathClaim failure – LUNz

October 16th, 2010
I stumbled across some interesting error messages the other day in a host’s vmkwarning log file. The errors were related to the deployment of PowerPath/VE and as I dug a little deeper I found this was an issue on all 3 hosts I had recently built.  When I connected via ILO I was seeing the additional error messages on the console as you can see from the screenshot below.
I was a bit concerned as this was my first deployment of ESX 4.1 with the latest PowerPath/VE 5.4 SP2 software.  I was also a little confused as I hadn’t actually presented any LUNs to this ESX cluster. All I had done was zone the HBA’s through to the storage processors and set up a storage group on the CLARiiON and added the hosts to it. So what path was PowerPath/VE trying to claim if there were no LUNs presented?
While on the service console of one host I ran the command esxcfg-mpath –l to list all the paths.  As you can see from the screenshot below I came across a fibre channel disk that the HBA’s was attempting to connect to, even though nothing had been presented.  I also noticed that it was using the VMware native multi-path plug-in and not PowerPath/VE and was also showing as dead! So was this the path that PowerPath was failing to claim?
Following discussions with EMC Support I was informed that this was what is known as a LUNz. This is a fake logical unit presented to the host by the storage processors to provide a path for host software to send configuration commands to the array when no LUNs have been presented.
In ESX 4.x the Navisphere host agent that you would normally install on a CLARiiON attached host is not supported.  Instead the ESX host registers itself with the storage array automatically (requires FLARE 28 or later). In order to do this it needs a communication path to the array, hence the LUN_Z being presented. If a LUN is presented to the host then that would be used instead,  therefore the LUNz disappear when a LUN is added as it is no longer required.
So I added a 100GB LUN to the storage group and I immediately received a number of email alerts from vCenter stating that connectivity had been lost to the naa.500 device.  So as expected as soon as LUN 0 is presented the LUN_Z is no longer required.
The key identifier that this was a LUNz was the naa.500 shown in the device name (see the screenshot above). A LUN presented from the CLARiiON shows up with a device name of naa.600 as you an see in the screenshot below.  You can also see now that the plug-in is now set to PowerPath which is what I would expect.
Looking in vCenter I can now see multiple active paths to the device which tells me that PowerPath/VE has truly kicked in and is working as expected.
There wasn’t a lot of information out there on the particular error messages I was receiving (listed below) hopefully this explanation will allay some fears.  I know I was concerned when I first saw the messages but thankfully it turned out to be by design.
EmcpEsxLogEvent: Error :emcp :MpxEsxPathClaim :Could not claim the path. Status : Failure
ScsiPath: 3815: Plugin ‘PowerPath’ had an error (Failure) while claiming path
ScsiDevice: 1726: Full GetDeviceAttributes during registration of device ‘naa.500…’  : failed with I/O error
ScsiDeiceIO: 5172: READ CAPACITY on device “naa.50…’ from Plugin “NMP” failed. I/O error

ESX, Storage, VMware , ,

HP Support Matrix – VMware / HP Downloads

September 30th, 2010

I’ve been building a few new HP DL 380 G7 servers as ESX 4.1 hosts and was having the usual nightmare finding anything on HP’s website. I was specifically looking for HP Insight Management Agent downloads and I’d been searching for ages when I came across a page called VMware from HP for Proliant. I’ve still no idea how I stumbled across it but I’m writing this down now so I never loose it.

The screenshot below which is from the bottom of the web page and lists all the HP Insight management agents for V4.x that I could possibly need.

As you can see from the next screenshot the web page covers all versions of ESX. It also has an extensive list of servers covering the BL, DL, ML and SL Proliant ranges.  Click on the tick and it will take you straight to the appropriate page for that server so you can download everything you may need regardless of end operating system. So it acts as a support matrix and also acts as a collection of links to the correct product download page, can’t ask for much more than that.  Good work HP, not often I say that about your web site.



ESX, ESXi, Servers, VMware , ,

Virtual Distributed Switch and vCenter Server failure

December 20th, 2009

I’m currently working with my colleagues on an upgrade of our VI 3.5 infrastructure to vSphere Enterprise Plus.  We have recently been mulling over some of the design elements we will have to consider and one of the ones that came up was virtual Distributed Switches (vDS).  We like the look of it,  it saves us having to configure multiple hosts with standard vSwitches and it also has some nice benefits such as enhanced network vMotion support, inbound and outbound traffic shaping and Private VLANs.

vDSOne of the questions that struck me was,  what happens if your vCenter server fails? what happens to your networking configuration? Surely your vCenter server couldn’t be a single point of failure for your virtual networking, could it?

Well I did a bit of digging about, chatted to a few people on twitter and the answer is no it would not result in a loss of virtual networking.  In vSphere vDS the switch is split into two distinct elements,  the control plane and the data plane. Previously both elements were host based and configured as such through connection to the host, either directly using the VI client or through vCenter. In vSphere because the control plane and data plane have been separated,  the control plane is now managed using vCenter only and the data plane remains host based.  Hence when your vCenter server fails the data plane is still active as it’s host based where as the control plane is unavailable as it’s vCenter based.

One thing I was not aware of was where all this vDS information is stored . Mike Laverick over at RTFM informed me that the central config for a vDS is stored on shared VMFS within a folder called the .dvsData folder. I’ve since learnt that this location is chosen automatically by vCenter and you can use the net-dvs command to determine that location. It will generally be on shared storage that all ESX hosts participating in the vDS have access to.  As a back up to this .dvsData folder a local database copy is located in /etc/vmware/dvsData.db which I imagine only comes into play if your vCenter server goes down or if your ESX host loses connectivity to the shared VMFS with the .dvsData folder.  You can read more about this over at RTFM

Interesting links if your considering VMware Distributed Switches

VMware’s demo video of vDS in action, for those who want to learn more about vDS

Mike Laverick’s great reasoning on whether you should use vDS or not

Eric Sloof’s vDS caveats and best practices article

VMware’s vSphere Networking white paper explaining new vDS features

VMware’s vSphere Networking white paper on vDS network migrations

Jason Boche’s very interesting article on a specific vCenter + vDS issue

ESX, vCenter ,

How to: Install QLogic HBA Libraries for HP Insight Manager on ESX 3.5 U4

November 8th, 2009

I’ve just finished a piece of work upgrading a few hosts to ESX 3.5 Update 4 using the ESX deployment appliance.  As part of the upgrade I install version 8.2.0 of the HP management agents for ESX. Upon completion of the install I realised that I could not see the Qlogic HBA’s within the Insight Manager home page.

Upon investigation it appears that HP have removed the libraries from their software from version 8.0.0 onwards.  Now ideally I would have installed the libraries before installing HP Insight Manager as indicated by Arnim Van Lieshout in his great HP agent upgrade script post.  In my case I had already done the install, so in order to see the HBA’s and the connected HP Storage, I had to carry out the following steps.

I downloaded the relevant Qlogic API library for ESX 3.5 U4 from the following site

I created a temporary folder in the /tmp partition called qlogiclib

I unzipped the contents and copied it over to the new folder using WinSCP.

I logged on to the service console and ran the following commands to stop the HP agents.

service hpsmhd stop and service hpasm stop

I then navigated to the /tmp/qlogiclib folder and ran the command ./

I then ran the following commands in the order shown below to restart the HP agents.

service hpsmhd start
and service hpasm start

To check if it worked I navigated to https://hostname:2381 and checked for the external storage connections entry under the storage section on the home page.

Finally I deleted the /tmp/qlogiclib folder as it was no longer required.

Now the same steps carried out with vSphere ESX 4 does not have the desired effect.  Even though you can use the HBA test script located in the Qlogic API library they will not show up in HP Insight manager (Much to my annoyance).  When I eventually find a solution I will of course provide an update.

ESX, VMware, vSphere ,

How to: Check and change the ESX Swap Partition

November 1st, 2009

An interesting problem occurred the other day with one of our older production ESX 3.0.2 hosts. For the first time with any ESX host we have the service console memory ran out,  this resulted in all VM’s becoming unresponsive and loss of service to our users.

Now these hosts were built a couple of years ago by a consultant and all had their service console memory set to the default value of 272MB. I’m in the process of upgrading all hosts to ESX 3.5 U4 and changing the memory levels to the maximum 800MB,  this particular host was due to be upgraded in the next 2 weeks.  Unfortunate timing!!

VMware support were as helpful as ever and informed my colleague to up the service console memory to 800MB.  My only concern was the fact that your swap space is meant to be twice your service console memory.  If the memory was only set to 272MB you can be sure that the swap partition wasn’t going to be set to 1600MB.

My colleague was having trouble finding out what size the swap partition was so I gave him a hand. First of all he was doing a df –k at the service console,  which shows him the named linux partitions but not the swap partition we were looking for.  To get information on all disks and partitions attached to the host we need to run fdisk – l

This command showed us the swap partition created was made up of 1044225 blocks, though we weren’t sure exactly what this equated to in MB.


I took a look at one of our newly built ESX 3.5 U4 hosts and compared it’s fdisk –l results to the scripts used to build it.  I quickly found that by dividing by 1024 you could get the size of the partitions.  So in this case the swap partition on the ESX 3.0.2 host was roughly 1GB which was less than the recommended 2 x console memory sizing.


On this occasion VMware support advised us that it should be OK as it was.  That coupled with the fact we are going to rebuild the server in the coming weeks was enough for us to call the case closed.

However what if we did want to change it? I’d always been taught that changing the swap partition after the host had been built usually meant a full rebuild.  However as I’ve been working my way through Scott Lowe’s Mastering Vmware vSphere 4 book I came across the steps to do it without a rebuild.  It’s always recommended to rebuild a host as opposed to take this action, however occasionally needs must.

first create a new swap file on an existing service console partition, the command below will create a 1.6GB within the path entered /path/to/

dd if=/dev/zero of=/path/to/swap.file bs=1024 count =1640144

Use the following command to turn this into a usable swap file

mkswap /path/to/swap.file

Now enable the swap file with the following command

swapon /path/to/swap.file

If you do try this, it is entirely at your own risk. I haven’t as I am planning to rebuild in the near future.  If I wasn’t I would probably have given this a shot just to put my mind at ease.

ESX, VMware, vSphere , , ,

Virtu-Al’s PowerShell VMware Daily Report

July 16th, 2009

For those of you that will have heard of Alan Renouf you will undoubtedly know of his talents in the dark art of VMware CLI / Powershell.  For those of you who don’t know him I suggest you check out his web site  to sample some of the many great articles and scripts he’s already produced.

His latest powershell creation has recieved a lot of attention in the last couple of days and with good reason. The Daily Report is a configurable script where you can set thresholds and variables such as snapshot age, datastore space free thresholds or number of days to look at for vCenter warnings and errors.  The script when run goes off and examines your Virtual  Infrastructure based on these variables and then proceeds to email you a nice html report on the following items.

·         VMs created in the  x number of days and who created them.

·         VMs deleted in the  x number of days and who deleted them.

·         Datastores which have less than x% of free space remaining.

·         VMs that have CD-Rom or Floppy drives connected.

·         VMs with no VMware Tools installed.

·         Snapshots that are older than x number of days.

·         Current state of vCenter Services.

·         vCenter events that have been logged in x number of days.

·         Windows events  on the vCenter server that relate to VMware.

·         Hosts in maintenance mode or a disconnected state.

Get yourself over  to Alan’s site and download a copy of the script and give it a try,  I did today and the results were enough for me to go ahead and implement this as a scheduled task.  If you’d like to see more features in Alan’s Daily Report script then give him some feedback,  there are a few good suggestions on the blog post already and I’m sure the next version isn’t far away.  Great work Alan, keep it up!

ESX, vCenter, VI Toolkit / Powershell, VMware , ,

EMC Configuresoft ESX compliance Checker

July 14th, 2009

I was purusing twitter the other day (as you do) when I came across a link shared by @Stevie_Chambers of VMware.  The link was for a new free tool from EMC called Compliance Checker for VMware ESX, which as you might guess does exactly what  it’s title suggests.  The tool though EMC branded is actually by Configuresoft who EMC bought outright earlier this year following an existing OEM relationship.

Now some of you may be familiar with the Tripwire Config Check tool which allows you to scan your ESX servers security configuration, which in itself is a very handy tool.  This new tool from EMC appears to be a little more comprehensive in that it checks your ESX servers against both the VMware VI3 Security Hardening Guidelines  and the Centre for Internet Security VMware ESX Server 3.x benchmark.  Test results against both are backed with an extensive knowledge base of what the check is and the actions required to rectify the issue.

Included below are some screenshots of the Main interface, the reports returned and the knowledgebase articles you recieve when you click on any item in the report.





I must admit that these compliance checkers are handy because my Linux experience only began with ESX and sometimes the areas of security being checked are ones I wouldn’t have a clue about.  This is where the knowledgebase is great because it explains a little about what the check actually is as well as the actions required to rectify the issue.  Very handy because it would appear that a standard build of ESX 3.5 U4 is only 73% compliant as far as this piece of software is concerned.

To get your free copy of Compliance Checker for VMware ESX, click the link,  you’ll need to register but it’s a small price to pay for this handy tool.

ESX, New Products, VMware ,

vSphere ESX4i on a USB key / Pen Drive

May 24th, 2009

As soon as vSphere was put on General release I downloaded a copy of vSphere 4 ESXi for running on my home lab setup.  I’ve only recently started my home lab and the first machine I purchased was the HP Proliant ML115 G5 server. This was following a recommendation by Kiwi Si over at  who has extensively blogged about the suitability of the HP ML110 G5 and ML115 G5 for ESX labs.

If your interested in starting a home lab I can thoroughly recommend these HP server.  Simon even has a deal going with (UK) for free delivery on either the HP ML110 or ML115 server.  Get over to his hot deals page for further details.

Simon also has some great articles on getting ESXi 3.5 running from a USB pen drive.  This was a perfect place for me to start as I wanted to take advantage of the ML115 G5′s internal USB port and boot my server from a USB pen drive. So where do you start?

For those of you using an Apple Mac and wanting to conduct this excercise, check out the following article over on Tom Rowan’s blog

What do I need?

A USB Pen Drive that is over 1GB in capacity – nice and cheap at amazon
Download the vSphere ESX4i ISO image from here 
Download Shareware version of Winrar from here
Download Free Trial version of Winimage from here

How do I do it?

Once you have downloaded the ISO image open it up with WinRAR,  make sure you use WinRAR as I had problems with WinZip and UltraISO


Double click on the image.tgz file to open the contents in WinRAR and drill down to the \usr\lib\vmware\installer directory.

Within this folder you will find a file called VMware-VMvisor-big-164009-x86_64.dd.bz2. This is another zip file so double click on it and the contents will be displayed in a seperate WinRAR window.


Once inside extract the file VMware-VMvisor-big-164009-x86_64.dd using WinRAR and copy it somewhere locally on your PC.

Now install and open up the WinImage trial that you downloaded at the start of this process.

Insert your USB key and then select Disk and restore virtual hard disk image on physical drive as per the screenshot below.


 Select the physical USB drive from the list and click OK,  when prompted for the virtual disk file navigate to the dd file you extracted to your local PC.  This will now image your USB Key with the vSphere ESX4i hypervisor.

Once complete stick the USB key in a server / whitebox that supports 64 bit computing and away you go.  The screenshot below shows my own HP Proliant ML115 G5 running vSphere ESX 4i and all this from a simple 2GB USB pen drive.


ESX, VMware, vSphere , ,

VMware Update Manager not downloading ESX updates

May 23rd, 2009

I’m currently working on replacing all our ESX 3.0.2 and 3.0.1 hosts with 3.5 U4 hosts,  unfortunately the leap to vSphere is going to have to wait until next year.

One of my main reasons for doing this was to allow for the introduction of VMware Update Manager for ESX Host patching.  ESX Hosting patching can be a bit of a pain when using the service console and Esxupdate and because of this patching starts to become a nightmare and gets forgotten about.

So I had my first ESX 3.5 host added to virtual center and was poking around the VMware Update Manager tab and noticed that there were absolutely no host patches attached to the default host baselines.  I did a bit of digging around online and plenty of people had problems out there with the actual installation of VUM or their vci-integrity.xml file was missing some of it’s content.

Luckily my issue was resolved far more easily.  My virtual center server didn’t appear to be able to access the websites to get the patches due to the enhanced security pack.  All I had to do was add the following websites into the trusted sites on Internet Explorer on the Virtual Center server.

When the next scheduled patch download occured I recieved 150 patches for ESX 3.5.0 and was then able to create a custom baseline for all patches after 3.5.0 U4 and apply them.  Worked an absolute treat.

ESX, ESXi, VMware ,

VMware vSphere – General Availability

May 21st, 2009

Today is the day, VMware is now available for download and some really good news is that VMware are offering a 60 day evaluation trial of vSphere Enterprise Plus and vCenter Server Standard.

Three links for you, the first is the VMware vSPhere Download page and the second is the free 60 day evaluation download link.  For those who want to use the free version of ESX 4i and have a computer / server capable of supporting 64 bit you can get your free copy at this link.

Documentation for the various vSphere components can be found here

Credit to Michael Hany over at for the links to these awesome video’s below, which should help you get to grips with some of the new vSphere 4 features.

ESX Installation and Configuration
ESXi Installation and Configuration
VMware vCenter Server
VMware vSphere Client
Networking configuration
Storage configuration (iSCSI)
Create and manage virtual machines
VMware Host Profiles
VMware Storage VMotion
VMware vCenter Server Linked Mode
VMware vNetwork Distributed Switch (vDS)

ESX, New Products, VMware, vSphere , ,