Posts Tagged ‘esx’

How to: Check and change the ESX Swap Partition

November 1st, 2009

An interesting problem occurred the other day with one of our older production ESX 3.0.2 hosts. For the first time with any ESX host we have the service console memory ran out,  this resulted in all VM’s becoming unresponsive and loss of service to our users.

Now these hosts were built a couple of years ago by a consultant and all had their service console memory set to the default value of 272MB. I’m in the process of upgrading all hosts to ESX 3.5 U4 and changing the memory levels to the maximum 800MB,  this particular host was due to be upgraded in the next 2 weeks.  Unfortunate timing!!

VMware support were as helpful as ever and informed my colleague to up the service console memory to 800MB.  My only concern was the fact that your swap space is meant to be twice your service console memory.  If the memory was only set to 272MB you can be sure that the swap partition wasn’t going to be set to 1600MB.

My colleague was having trouble finding out what size the swap partition was so I gave him a hand. First of all he was doing a df –k at the service console,  which shows him the named linux partitions but not the swap partition we were looking for.  To get information on all disks and partitions attached to the host we need to run fdisk – l

This command showed us the swap partition created was made up of 1044225 blocks, though we weren’t sure exactly what this equated to in MB.


I took a look at one of our newly built ESX 3.5 U4 hosts and compared it’s fdisk –l results to the scripts used to build it.  I quickly found that by dividing by 1024 you could get the size of the partitions.  So in this case the swap partition on the ESX 3.0.2 host was roughly 1GB which was less than the recommended 2 x console memory sizing.


On this occasion VMware support advised us that it should be OK as it was.  That coupled with the fact we are going to rebuild the server in the coming weeks was enough for us to call the case closed.

However what if we did want to change it? I’d always been taught that changing the swap partition after the host had been built usually meant a full rebuild.  However as I’ve been working my way through Scott Lowe’s Mastering Vmware vSphere 4 book I came across the steps to do it without a rebuild.  It’s always recommended to rebuild a host as opposed to take this action, however occasionally needs must.

first create a new swap file on an existing service console partition, the command below will create a 1.6GB within the path entered /path/to/

dd if=/dev/zero of=/path/to/swap.file bs=1024 count =1640144

Use the following command to turn this into a usable swap file

mkswap /path/to/swap.file

Now enable the swap file with the following command

swapon /path/to/swap.file

If you do try this, it is entirely at your own risk. I haven’t as I am planning to rebuild in the near future.  If I wasn’t I would probably have given this a shot just to put my mind at ease.

ESX, VMware, vSphere , , ,

Virtu-Al’s PowerShell VMware Daily Report

July 16th, 2009

For those of you that will have heard of Alan Renouf you will undoubtedly know of his talents in the dark art of VMware CLI / Powershell.  For those of you who don’t know him I suggest you check out his web site  to sample some of the many great articles and scripts he’s already produced.

His latest powershell creation has recieved a lot of attention in the last couple of days and with good reason. The Daily Report is a configurable script where you can set thresholds and variables such as snapshot age, datastore space free thresholds or number of days to look at for vCenter warnings and errors.  The script when run goes off and examines your Virtual  Infrastructure based on these variables and then proceeds to email you a nice html report on the following items.

·         VMs created in the  x number of days and who created them.

·         VMs deleted in the  x number of days and who deleted them.

·         Datastores which have less than x% of free space remaining.

·         VMs that have CD-Rom or Floppy drives connected.

·         VMs with no VMware Tools installed.

·         Snapshots that are older than x number of days.

·         Current state of vCenter Services.

·         vCenter events that have been logged in x number of days.

·         Windows events  on the vCenter server that relate to VMware.

·         Hosts in maintenance mode or a disconnected state.

Get yourself over  to Alan’s site and download a copy of the script and give it a try,  I did today and the results were enough for me to go ahead and implement this as a scheduled task.  If you’d like to see more features in Alan’s Daily Report script then give him some feedback,  there are a few good suggestions on the blog post already and I’m sure the next version isn’t far away.  Great work Alan, keep it up!

ESX, vCenter, VI Toolkit / Powershell, VMware , ,

EMC Configuresoft ESX compliance Checker

July 14th, 2009

I was purusing twitter the other day (as you do) when I came across a link shared by @Stevie_Chambers of VMware.  The link was for a new free tool from EMC called Compliance Checker for VMware ESX, which as you might guess does exactly what  it’s title suggests.  The tool though EMC branded is actually by Configuresoft who EMC bought outright earlier this year following an existing OEM relationship.

Now some of you may be familiar with the Tripwire Config Check tool which allows you to scan your ESX servers security configuration, which in itself is a very handy tool.  This new tool from EMC appears to be a little more comprehensive in that it checks your ESX servers against both the VMware VI3 Security Hardening Guidelines  and the Centre for Internet Security VMware ESX Server 3.x benchmark.  Test results against both are backed with an extensive knowledge base of what the check is and the actions required to rectify the issue.

Included below are some screenshots of the Main interface, the reports returned and the knowledgebase articles you recieve when you click on any item in the report.





I must admit that these compliance checkers are handy because my Linux experience only began with ESX and sometimes the areas of security being checked are ones I wouldn’t have a clue about.  This is where the knowledgebase is great because it explains a little about what the check actually is as well as the actions required to rectify the issue.  Very handy because it would appear that a standard build of ESX 3.5 U4 is only 73% compliant as far as this piece of software is concerned.

To get your free copy of Compliance Checker for VMware ESX, click the link,  you’ll need to register but it’s a small price to pay for this handy tool.

ESX, New Products, VMware ,

VMware vSphere – General Availability

May 21st, 2009

Today is the day, VMware is now available for download and some really good news is that VMware are offering a 60 day evaluation trial of vSphere Enterprise Plus and vCenter Server Standard.

Three links for you, the first is the VMware vSPhere Download page and the second is the free 60 day evaluation download link.  For those who want to use the free version of ESX 4i and have a computer / server capable of supporting 64 bit you can get your free copy at this link.

Documentation for the various vSphere components can be found here

Credit to Michael Hany over at for the links to these awesome video’s below, which should help you get to grips with some of the new vSphere 4 features.

ESX Installation and Configuration
ESXi Installation and Configuration
VMware vCenter Server
VMware vSphere Client
Networking configuration
Storage configuration (iSCSI)
Create and manage virtual machines
VMware Host Profiles
VMware Storage VMotion
VMware vCenter Server Linked Mode
VMware vNetwork Distributed Switch (vDS)

ESX, New Products, VMware, vSphere , ,

vSphere 4.0 – What’s new in vSphere Storage

May 17th, 2009

This weekend I finally had the chance to catchup on some of the new storage features released as part of vSphere 4.0,  there are quite a few changes to cover,  some of them quite exciting.

VMFS Upgrade

Once of the good pieces of news to come out is that the VMFS changes in vSphere are minimal.  vSphere 4.0 introduces a minor point release (3.3.0 to 3.3.1) with some subtle changes,  so much so that it’s not really been documented anywhere.  Most of the changes with VMFS are actually delivered within the VMFS driver at the VMKernel level,  this is where most of the I/O improvements and features such as thin provisioning have been delivered as part of vSphere.

Upgrading VMFS was a major step in the upgrade from VMFS 2 to VMFS 3,  good to hear that there are no major drivers to upgrade VMFS as part of your vSphere upgrade.  Any new VMFS datastores created with the new vSphere hosts will of course be VMFS 3.3.1 however this is backwardly compatible with earlier versions of ESX 3.x.  If you really want to move onto the new version of VNFS, format some new datastores and use Storage vMotion to move your VM’s onto the new VMFS 3.3.1 datastores. 

Thin Provisioning

Thin provisioning is one of the areas that excites me most about the new vSphere release.  I conducted a very quick survey of my employers development and system test ESX environments recently and found that currently we were only utilising 48% of virtual storage that had been provisioned.  It’s easy to see where immediate savings can be made simply by implementing vSphere and thin provisioning.  I’ll be using that in the cost benefits case for sure!

Thin provisioning is nothing new,  it has been available at the array level for a while now, so one of the big questions is where should I thin provision?  Well that really depends what kind of environment you have I suppose.  Smaller customers will benefit greatly from VMware thin provisioning as they probably don’t own arrays capable of TP.  Bigger companies on the other hand might well benefit from carrying out both as they have both the skill sets and the equipment to full utilise it at both levels. 

Chad Sakac has written a superb article entitled “thin on thin where should you do thin provisioning vsphere 4.0 or array level” which goes deep into the new thin provisioning features and the discussions around what’s the best approach. I strongly suggest people give it a read,  it explains pretty much all you need to know.

Storage VMotion

The Storage vMotion in ESX 3.5 had a few limitations which vSphere addresses.  It’s now fully integrated with vCenter as opposed to being command line based in the previous version,  it allows for the moving of a VM between different storage types, i.e. FC, ISCSI or NFS.  One excellent usage of Storage vMotion is the ability to migrate your thick vm’s and change them to thin VM’s.  Perfect for reclaiming disk space and increasing utilisation without downtime, brilliant!
Storage vMotion has also been enhanced from an operational perspective. Previously storage vmotion involved taking a snapshot of a disk,  copying the parent disk to it’s new location and then taking the child snapshot and re-parenting the child disk with the parent.  This process required the 2 x the CPU and memory of the VM being migrated in order to ensure zero downtime.  In vSphere 4.0 Storage vMotion uses change block tracking and a process very similar to how vMotion deals with moving active memory between hosts.  The new Storage vMotion conducts an iterative process scanning what blocks have been changed, each iterative scan should result in smaller and smaller increments and when it gets down to a small enough size it conducts a very quick suspend / resume operation as opposed to using the doubling up resources method that it previously needed to.  Making it faster and more efficient than it was in it’s previous incarnation.

Para Virtualised SCSI

Para Virtualised SCSI (PVSCSI) is a new driver for I/O intensive virtual machines. VMware compare this to the vmxnet adapter,  which is an enhanced and optimised network driver providing higher performance.  PVSCSI is similar, it’s a specific driver that offers higher I/O throughput, lower latency and lower CPU utilisation within virtual machines. Figures discussed by Paul Manning on the recent Vmware community podcast included 92% increase in IOPS throughput and 40% decrease in latency when compared to the standard LSI / BUSLogic virtual driver.

A caveat of this technology is that the guest OS still has to boot from a non PVSCSI adapter (LSI / Buslogic),  you would look to add your PVSCSI adapter for your additional data virtual disks.  Currently only Windows 2003, Windows 2008 and RH Linux 5 have the software drivers to take adavantage of this new adapter.

Update  - Chad Sakac has posted a new EMCWorld I/O Performance comparison of the vSphere PVSCSI adpater vs the LSI SCSI adapter, check out the link for more details.

VMware Storage Book

Paul Manning mentioned on the recent podcast that VMware are planning a book dedicated to Virtualisation and storage in an attempt to consolidate the amount of documentation out there on Storage configuration and best practice.  Currently users need to look through 600 pages of the SAN Config guide and vendor guidelines. VMware would hope to try boil this down to a much more manageable 100 – 150 pages.

If you can’t wait that long, Chad Sakac has written the storage chapter in Scott Lowe’s new vSphere book which I believe is available for pre-order on Amazon

vSphere Storage WhitePaper

Paul Manning who I’ve mentioned in this blog post has written a great 10 page white paper explaining all of these features in more detail along with some of the more experimental features I haven’t mentioned.

Gestalt-IT, New Products, Storage, VMware, vSphere , , , ,

Good Deal & Free Delivery on HP Proliant ML110 & ML115 G5

March 19th, 2009

I’ve been considering purchasing a server for some time,  so I can setup a home lab for training purposes.  I’ve been reading about the HP Proliant ML110 G5 and ML115 G5 make good ESX hosts in home lab environments.  I had a conversation with Simon over at recently about how the price of these servers had shot up from a very appealing £120, to well over £350. I was kind of facing up to paying the extra,  but Simon informed me he had a deal brewing regarding these servers and to hold on.

Well he’s now posted the details on how to get these servers for £199, ex VAT and get free delivery thrown in. I’ve personally got my eye on the Quad Core AMD Opteron based ML115 G5,  it only comes with 512MB of memory however I found 4GB for it at a cost of £45.99 from Crucial memory. For another £28 you can add another HP NIC

Get on over to his blog post for further details, take advantage of this deal and get your home lab up and running without delay.  Good Deal & Free Delivery on HP Proliant ML110 & ML115 G5

General ,

Improving Vmotion Performance

January 5th, 2009

I follow a lot of people on twitter who write about virtualisation and have there own blog.  So up popped a message from Jason Boche over at about a new article he’d put up on the site about Vmotion performance.

Written by a gentleman called Simon Long it is a excellent article.  It just goes to shows what happens when something doesn’t work as quickly as you’d like,  some people have the determination and spend the time finding out why.  This post is a gem and one of those little tweaks that will come in very handy in the future

I’ll set the scene a little….

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, *********, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.

Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.

1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpdx.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;


5. Now you need to decide what value to give “maxCostPerHost”.

A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).

I am unsure on the maximum value that you can use here, the largest I tested was 24.

6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.

Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.

I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.

Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.

The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;

Section 1 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30

Section 2 – This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).

Section 3 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36

Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.

Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24

Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54

Time Different = 30secs

It’s still an improvement all be it not so big.

Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.

I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.

ESX, vCenter, VMware ,

Free ESX Host security analysis tool – Tripwire Config Check

January 2nd, 2009

Some time ago long before I started my blog I used an application called Trip Wire Config Check on some of my employers ESX Hosts. We are primarily a Microsoft software house so our technical experience of Linux / Unix was minimal before the introduction of ESX into our server estate. The config check application checks the current host configuration against the VI3 Security hardening guidelines reporting back any vunerabilities.

A some of the configuration parameters being checked include:

- Virtual network labeling
- Port Group settings
- Network isolation for VMotion and iSCSI
- NIC Mode settings / Layer 2 Security settings
- VMWare ESX Service Console security settings
- SAN resource masking and zoning

The results that the tripwire config check returned showed that we had a number of areas that needed to be addressed with our host security configurations. The results provided a great checklist of everything that was right and wrong with our current build procedures.  Some of the items that came up we would never have even considered,  some of them we felt were minor and could be ignored,  others needed to be addressed immediately.  I highly recommend taking a look at this free tool and running it against one of your standard host builds to see if you need to change it, hopefuly you won’t need to revist all your hosts.

Download your free copy of Tripwire – Config Check and check out the following blog post for installation and usage instructions.  

Tripwire® ConfigCheckTM is a free utility that rapidly assesses the security of VMware ESX 3.0 and 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

ESX, New Products, VMware ,

VMware SRM (Site Recovery Manager) Demo Video

December 26th, 2008

I’ve recently been looking at posting something on SRM as there seems to be so much information out there. I will hopefully find the time to write a post that summarises everything I’ve seen so far but I thought this would be a good starter for ten.  Richard Garsthagen over at run virtual has posted a new official VMware video introduction to VMware Site Recover Manager (SRM),  You can view it below.

He also included a handy link to a document which tells you how to setup a full 2 site SRM demo suite with ESX on a single laptop.  Handy for setting up a demo lab if you want to get familiar with the product or if your’re a VMware partner you may want to set this up for a sales demonstration.

New Products, VMware , ,

VMFS Heap Size error messages

December 24th, 2008

I subscribe to a large number of virtualisation feeds and the primary reason is that I never fail to learn something new every single day.

I came across a blog entry by the excellent Duncan Epping over at Yellow Bricks about VMFS Heap size configuration.  I was not previously aware of this setting because we have 3.0.x in our environment and our HP MSA storage only support 3.6Gb fully populated so this scenario would never come up.  I will however be keeping this in mind when designing our new ESX 3.5 environment.  We seem to be utilising bigger hosts all the time and when you add in a new EMC Clariion CX4, the potential for one host to be accessing more than 4TB of open VMDKs goes up.

was talking to a fellow consultant today. He ran into the following error messages at one of his customer sites:

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0×8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

During the conversation I knew I’d seen this problem before. But the problem that I witnessed was related to a high threshold value in Vizioncore vFoglight. I knew it was possible to change the setting:

  1. Open vCenter, and click a specific host
  2. Click on the “Configurations” tab
  3. Click on Advanced Settings, VMFS3
  4. Change the value of “VMFS3.MaxHeapSizeMB”


The default value is 16MB, this allows for a maximum of 4TB of open vmdk’s on a single host. The max setting is 128MB which allows for a maximum of 32TB of open vmdk’s on a single host. Keep this in mind when designing your environment.

Keep in mind that this is ESX 3.5 only, you can’t change the heap size in ESX 3.0.x.

ESX, VMware , ,