VirtualPro

An interesting problem occurred the other day with one of our older production ESX 3.0.2 hosts. For the first time with any ESX host we have the service console memory ran out,  this resulted in all VM’s becoming unresponsive and loss of service to our users.

Now these hosts were built a couple of years ago by a consultant and all had their service console memory set to the default value of 272MB. I’m in the process of upgrading all hosts to ESX 3.5 U4 and changing the memory levels to the maximum 800MB,  this particular host was due to be upgraded in the next 2 weeks.  Unfortunate timing!!

VMware support were as helpful as ever and informed my colleague to up the service console memory to 800MB.  My only concern was the fact that your swap space is meant to be twice your service console memory.  If the memory was only set to 272MB you can be sure that the swap partition wasn’t going to be set to 1600MB.

My colleague was having trouble finding out what size the swap partition was so I gave him a hand. First of all he was doing a df –k at the service console,  which shows him the named linux partitions but not the swap partition we were looking for.  To get information on all disks and partitions attached to the host we need to run fdisk – l

This command showed us the swap partition created was made up of 1044225 blocks, though we weren’t sure exactly what this equated to in MB.

I took a look at one of our newly built ESX 3.5 U4 hosts and compared it’s fdisk –l results to the scripts used to build it.  I quickly found that by dividing by 1024 you could get the size of the partitions.  So in this case the swap partition on the ESX 3.0.2 host was roughly 1GB which was less than the recommended 2 x console memory sizing.

On this occasion VMware support advised us that it should be OK as it was.  That coupled with the fact we are going to rebuild the server in the coming weeks was enough for us to call the case closed.

However what if we did want to change it? I’d always been taught that changing the swap partition after the host had been built usually meant a full rebuild.  However as I’ve been working my way through Scott Lowe’s Mastering Vmware vSphere 4 book I came across the steps to do it without a rebuild.  It’s always recommended to rebuild a host as opposed to take this action, however occasionally needs must.

first create a new swap file on an existing service console partition, the command below will create a 1.6GB within the path entered /path/to/

dd if=/dev/zero of=/path/to/swap.file bs=1024 count =1640144

Use the following command to turn this into a usable swap file

mkswap /path/to/swap.file

Now enable the swap file with the following command

swapon /path/to/swap.file

If you do try this, it is entirely at your own risk. I haven’t as I am planning to rebuild in the near future.  If I wasn’t I would probably have given this a shot just to put my mind at ease.

ESX, VMware, vSphere esx, Service Console, Swap partition, VSphere

I recently had a number of vSphere ESX4i  USB Key installs following my article on putting vSphere ESX4i on a USB key / Pen Drive. I needed to format a couple for general windows usage, only to find that the ESX4i image creates a number of partitions on the USB Key. Unfortunately Windows does not appear to support the removal of partitions on removable devices so when I was trying to format a 2GB USB stick I was able to format a 110MB partition and that was it. I was a bit stuck on the best way to rectify the issue and wasn’t finding much to help out on the web.

That’s when I stumbled upon the HP USB Storage Format Tool,  a great little tool that works with a wide range of USB sticks and not just HP ones.  It allowed me to wipe the USB key as a single entity and didn’t care about the partitioning, returning my USB Key to a useable state within windows.

You can download it from HP’s website by clicking on this link,  sometimes you just don’t know if you can trust other download sites.

General ESX 3.5i, esx4i, USB, VSphere

I’ve recently been evaluating some of the new features in VMware vSphere to see what use they would be to my current employer. One of the areas that I touched upon in my “what’s new in vSphere Storage”  blog post was thin provisioning.  I wanted to come back and cover this particular topic in more detail as it’s a key feature and it’s available throughout all versions of vSphere so I’m sure everyone will be interested in it.

What is Thin Provisioning?

Thin Provisioning in it’s simplest form is only using the disk space you need.  Traditionally with virtual machines if you create a 500GB virtual disk it will use 500GB of your VMFS datastore. With Thin Provisioning you can create a 500GB virtual disk, but if only 100GB is in use only 100GB of your VMFS datastore will be utilised. Credit to Chad Sakac for the diagrams below.

How does it work in vSphere?

Thin Provisioning is being heralded as something new with vSphere,  when in truth it was already available in VI3.  In VI3 creating a thin provisioned disk involved using vmkfstools and was also not a production supported VM configuration.  Now in vSphere the creation of thin provisioned disks can be carried out from the VI Client (see below) and is a supported production configuration for a VM.

It’s as simple as checking a check box, the results are pretty good to.  Below you can see I have created two thin provisioned VM’s on my new ESX4i host and you can see the provisioned space and the used space stats being shown in the VI Client. 

What are the benefits?

The thin provisioning  feature is perfect for my home lab environment where disk space is at a premium, but how does it translate into real world implementations of ESX.  Well I for one have been looking at exactly this to identify what benefits could be achieved within my employers ESX estate. A quick audit found that our development and system test ESX environment was running at 48% disk utilisation,  so straight away thin provisioning would save us 52% on storage capacity used. Paul Manning of VMware mentioned on a recent communities podcast that on average vSphere would save users 50% on storage.   This is possibly not such a big thing when your talking about test environments, but when you move up to production SAN Storage, saving 50% on an expensive SAN array is a very real and tangabile cost saving.  One that people should definately take into account when making a cost benefit case for buying or upgrading to vSphere.

What are the potential downsides?

One of my personal concerns with thin provisioning is the potential overhead on any write activity that would requires the extension of the VMDK file.  To me there is an obvious VMFS operation that needs to take place there which would add to the overall time to complete the disk write.  When there is a requirement to expand a disk, the VMDK files will increase in increments based on the block size of the underlying VMFS partition, 1MB, 2MB, 4MB or 8MB.  So the overhead may be smaller if your VMFS has been formatted with a bigger block size, i.e. for a 16MB write it only has to expand 2 blocks when the VMFS block size is 8MB but would have to expand 16 blocks if it was formatted with the 1MB block size.  I can imagine this percieved overhead could put people off using thin provisioned disks for certain production based environments, especially those where there is a lot of write I/O activity,  SQL Server or Exchange for example.  To counteract that though,  the improvements in the VMware I/O Stack should compensate for this performance overhead.  This could potentially leave you in a situation where you’ve reduced a VM’s storage footprint and still have performance equal to that experienced in VI3,  possibly not a bad trade off.  I’d also expect people running their VMware environment on enterprise SAN technologies from the likes of EMC or NetApp to notice minimal performance impact with thin provisioning as SAN memory caches help take up the strain.

Another downside is if you want to use VMware Fault Tolerance to protect a VM then you cannot use thin provisioned disks.  To be honest this is a small issue as Fault Tolerance protection is most likely going to be on virtual machines that are important to your organisation.  These machines are probably the ones you wouldn’t thin provision in the first place for performance reasons.

Thin provisioning creates it’s own unique problem in that what we’re basically doing here is over provisioning the storage.  You need to keep a very close eye on thin provisioning as it’s quite feasabile that your VMFS datastore could fill up and your virtual machines fall over.  Not what you want to come into on a Monday morning,  or any morning for that matter.  So you need to monitor your storage and ensure that there is enough free space.  One of the simplest ways to do this is through the use of the new alarms in vSphere that allow you to alert on datastore usage and datastore over provisioning.  These should keep you from filling a datastore and killing your VM’s or ESX Servers

One gotcha that you should watch out for is VM swap files, as these are usually stored with your virtual machines vmdk files in the VMFS datastore.  In VI3 the swap file was not deleted when a VM was powered down,  in vSphere the swap file is deleted on power down and recreated when the VM is powered up.  You should be aware of this when over provisioning storage as you could get into a situation whereby you find you can’t power on a VM because there isn’t enough space for the swap file to be created.  This becomes more likely as servers and VM configuration maximum’s increase,  if you have a VM with 20GB of RAM it’s going to need 20GB of disk space for the swap file.  if you have 256GB of RAM in your vSphere host and you allocate it all out to VM’s then you need to think about the 256GB of disk capacity required to service virtual machine swap files.

Storage vMotion

If you’ve already got a VI3 environment then the chances are that your VM’s aren’t thin provisioned,  how on earth are you going to take advantage of this new feature? Well if you have purchased a vSphere edition that supports storage vMotion then you can of course migrate the underlying storage and have it thin provisioned during the move.  This should allow existing VI3 customers to claim back a lot of space,  as I mentioned before I found that our development and test VI environments were only 48% utilised.  If I storage vmotion all those VM’s and thin provision at the same time I will free up about 1.5TB of storage that wasn’t being used in the first place.

I’ve included a video below which demonstrates the Storage vMotion and thin provisoning features in vSphere quite nicely, enjoy!

Gestalt-IT, Storage, VMware, vSphere Storage vMotion, Thin Provisioning, VSphere

Well fresh from my return from the Citrix iForum I decided to fire head long into installing XenServer in my home lab so I could have a look at it.

I already run VMware vSphere 4i on my home lab which consists of an HP Proliant ML115 G5.  Instead of buying another machine to install Xenserver on or rebuilding my current vSphere server I thought I would try and install XenServer inside a virtual machine.  As Eric Gray over at vCritical proved you can install vSphere 4 inside a vSphere 4 virtual machine so surely the same would be possible XenServer 5.5, shouldn’t it?

Well the screenshot below should prove exactly that,  Xenserver 5.5 successfully running on vSphere 4i

So how did I conduct the install,  well first of all I downloaded the ISO from Citrix’s website and then did the following.

1 - Create a Virtual Machine with custom settings.
2 - Select the new Virtual Machine version 7 hardware.
3 - Select Red Hat Enterprise Linux v5 (64-bit).
4 - 1 vCPU and 1GB of RAM will suffice.
5 - I used the LSI Logic Parallel SCSI Controller.
6 - Create a disk based between 20 – 30GB (less has caused issues, see comments below)
7 - Make it thin provisioned if you want,  why wouldn’t you?
8 - Connect the ISO image to the VM and start it.
9 - Follow the prompts on screen to complete the install.

I only had one issue during install and that was when the following message appeared,  I carried on installing XenServer and it completed without issue.

However when It came to starting up windows based Virtual Machines,  like the message above indicated, I couldn’t.  XenCenter showed the following error.

Basically because Windows requires the hardware virtualised assist features (Intel VT or AMD-V),  hypervisor on top of hypervisor masks this underlying virtualisation assistance and hence Windows can’t operate.  What I did manage to get up and running was virtual machines running Debian Lenny 5.0,  so at least I had something to play about with and test out XenServer features such as live motion. Linux machines on XenServer start up in a para virtualised mode and are therefore supported where hardware virtualisation assist is not available.

check out the Debian Lenny based DreamLinux desktop edition,  this should give you some VM’s to play within your virtualised XenServer environment.

So although I didn’t get XenServer operating like I wanted to in Vmware vSphere, I did get  it working enough to play about with it and it’s features.  To be honest that’s all I was after in the first place!!

Citrix, VMware, XenServer, vSphere Citrix, VMware, VSphere, XenServer

As soon as vSphere was put on General release I downloaded a copy of vSphere 4 ESXi for running on my home lab setup.  I’ve only recently started my home lab and the first machine I purchased was the HP Proliant ML115 G5 server. This was following a recommendation by Kiwi Si over at www.techhead.co.uk  who has extensively blogged about the suitability of the HP ML110 G5 and ML115 G5 for ESX labs.

If your interested in starting a home lab I can thoroughly recommend these HP server.  Simon even has a deal going with ServersPlus.com (UK) for free delivery on either the HP ML110 or ML115 server.  Get over to his hot deals page for further details.

Simon also has some great articles on getting ESXi 3.5 running from a USB pen drive.  This was a perfect place for me to start as I wanted to take advantage of the ML115 G5’s internal USB port and boot my server from a USB pen drive. So where do you start?

For those of you using an Apple Mac and wanting to conduct this excercise, check out the following article over on Tom Rowan’s blog

What do I need?

A USB Pen Drive that is over 1GB in capacity - nice and cheap at amazon
Download the vSphere ESX4i ISO image from here 
Download Shareware version of Winrar from here
Download Free Trial version of Winimage from here

How do I do it?

Once you have downloaded the ISO image open it up with WinRAR,  make sure you use WinRAR as I had problems with WinZip and UltraISO

Double click on the image.tgz file to open the contents in WinRAR and drill down to the \usr\lib\vmware\installer directory.

Within this folder you will find a file called VMware-VMvisor-big-164009-x86_64.dd.bz2. This is another zip file so double click on it and the contents will be displayed in a seperate WinRAR window.

Once inside extract the file VMware-VMvisor-big-164009-x86_64.dd using WinRAR and copy it somewhere locally on your PC.

Now install and open up the WinImage trial that you downloaded at the start of this process.

Insert your USB key and then select Disk and restore virtual hard disk image on physical drive as per the screenshot below.

 Select the physical USB drive from the list and click OK,  when prompted for the virtual disk file navigate to the dd file you extracted to your local PC.  This will now image your USB Key with the vSphere ESX4i hypervisor.

Once complete stick the USB key in a server / whitebox that supports 64 bit computing and away you go.  The screenshot below shows my own HP Proliant ML115 G5 running vSphere ESX 4i and all this from a simple 2GB USB pen drive.

ESX, VMware, vSphere esx4i, esxi, VSphere

Today is the day, VMware is now available for download and some really good news is that VMware are offering a 60 day evaluation trial of vSphere Enterprise Plus and vCenter Server Standard.

Three links for you, the first is the VMware vSPhere Download page and the second is the free 60 day evaluation download link.  For those who want to use the free version of ESX 4i and have a computer / server capable of supporting 64 bit you can get your free copy at this link.

Documentation for the various vSphere components can be found here http://www.vmware.com/support/pubs/vs_pubs.html

Credit to Michael Hany over at www.hypervizor.com for the links to these awesome video’s below, which should help you get to grips with some of the new vSphere 4 features.

ESX Installation and Configuration
ESXi Installation and Configuration
VMware vCenter Server
VMware vSphere Client
Networking configuration
Storage configuration (iSCSI)
Create and manage virtual machines
VMware Host Profiles
VMware Storage VMotion
VMware vCenter Server Linked Mode
VMware vNetwork Distributed Switch (vDS)

ESX, New Products, VMware, vSphere esx, esxi, VSphere

This weekend I finally had the chance to catchup on some of the new storage features released as part of vSphere 4.0,  there are quite a few changes to cover,  some of them quite exciting.

VMFS Upgrade

Once of the good pieces of news to come out is that the VMFS changes in vSphere are minimal.  vSphere 4.0 introduces a minor point release (3.3.0 to 3.3.1) with some subtle changes,  so much so that it’s not really been documented anywhere.  Most of the changes with VMFS are actually delivered within the VMFS driver at the VMKernel level,  this is where most of the I/O improvements and features such as thin provisioning have been delivered as part of vSphere.

Upgrading VMFS was a major step in the upgrade from VMFS 2 to VMFS 3,  good to hear that there are no major drivers to upgrade VMFS as part of your vSphere upgrade.  Any new VMFS datastores created with the new vSphere hosts will of course be VMFS 3.3.1 however this is backwardly compatible with earlier versions of ESX 3.x.  If you really want to move onto the new version of VNFS, format some new datastores and use Storage vMotion to move your VM’s onto the new VMFS 3.3.1 datastores. 

Thin Provisioning

Thin provisioning is one of the areas that excites me most about the new vSphere release.  I conducted a very quick survey of my employers development and system test ESX environments recently and found that currently we were only utilising 48% of virtual storage that had been provisioned.  It’s easy to see where immediate savings can be made simply by implementing vSphere and thin provisioning.  I’ll be using that in the cost benefits case for sure!

Thin provisioning is nothing new,  it has been available at the array level for a while now, so one of the big questions is where should I thin provision?  Well that really depends what kind of environment you have I suppose.  Smaller customers will benefit greatly from VMware thin provisioning as they probably don’t own arrays capable of TP.  Bigger companies on the other hand might well benefit from carrying out both as they have both the skill sets and the equipment to full utilise it at both levels. 

Chad Sakac has written a superb article entitled “thin on thin where should you do thin provisioning vsphere 4.0 or array level” which goes deep into the new thin provisioning features and the discussions around what’s the best approach. I strongly suggest people give it a read,  it explains pretty much all you need to know.

Storage VMotion

The Storage vMotion in ESX 3.5 had a few limitations which vSphere addresses.  It’s now fully integrated with vCenter as opposed to being command line based in the previous version,  it allows for the moving of a VM between different storage types, i.e. FC, ISCSI or NFS.  One excellent usage of Storage vMotion is the ability to migrate your thick vm’s and change them to thin VM’s.  Perfect for reclaiming disk space and increasing utilisation without downtime, brilliant!
 
Storage vMotion has also been enhanced from an operational perspective. Previously storage vmotion involved taking a snapshot of a disk,  copying the parent disk to it’s new location and then taking the child snapshot and re-parenting the child disk with the parent.  This process required the 2 x the CPU and memory of the VM being migrated in order to ensure zero downtime.  In vSphere 4.0 Storage vMotion uses change block tracking and a process very similar to how vMotion deals with moving active memory between hosts.  The new Storage vMotion conducts an iterative process scanning what blocks have been changed, each iterative scan should result in smaller and smaller increments and when it gets down to a small enough size it conducts a very quick suspend / resume operation as opposed to using the doubling up resources method that it previously needed to.  Making it faster and more efficient than it was in it’s previous incarnation.

Para Virtualised SCSI

Para Virtualised SCSI (PVSCSI) is a new driver for I/O intensive virtual machines. VMware compare this to the vmxnet adapter,  which is an enhanced and optimised network driver providing higher performance.  PVSCSI is similar, it’s a specific driver that offers higher I/O throughput, lower latency and lower CPU utilisation within virtual machines. Figures discussed by Paul Manning on the recent Vmware community podcast included 92% increase in IOPS throughput and 40% decrease in latency when compared to the standard LSI / BUSLogic virtual driver.

A caveat of this technology is that the guest OS still has to boot from a non PVSCSI adapter (LSI / Buslogic),  you would look to add your PVSCSI adapter for your additional data virtual disks.  Currently only Windows 2003, Windows 2008 and RH Linux 5 have the software drivers to take adavantage of this new adapter.

Update  - Chad Sakac has posted a new EMCWorld I/O Performance comparison of the vSphere PVSCSI adpater vs the LSI SCSI adapter, check out the link for more details.

VMware Storage Book

Paul Manning mentioned on the recent podcast that VMware are planning a book dedicated to Virtualisation and storage in an attempt to consolidate the amount of documentation out there on Storage configuration and best practice.  Currently users need to look through 600 pages of the SAN Config guide and vendor guidelines. VMware would hope to try boil this down to a much more manageable 100 - 150 pages.

If you can’t wait that long, Chad Sakac has written the storage chapter in Scott Lowe’s new vSphere book which I believe is available for pre-order on Amazon

vSphere Storage WhitePaper

Paul Manning who I’ve mentioned in this blog post has written a great 10 page white paper explaining all of these features in more detail along with some of the more experimental features I haven’t mentioned. 

http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereStorage_P10_R1.pdf

Gestalt-IT, New Products, Storage, VMware, vSphere esx, Storage, Storage vMotion, Thin Provisioning, VSphere

Following the completion of my post last night about the potential paths that the VMware certification was going to take,  VMware have made an official announcement on the vSphere VCP.

To see the official VMware VCP certification path diagram click the link

For more information check out the following VMware education services page. This confirms that existing VCP’s can sit the exam without sitting the “what’s new in vSphere” course, which is great news !!

Certifications, VMware, vSphere Certifications, VMware, VSphere

Been busy putting together a case on why its a good idea to take advantage of VMware’s current vSphere  upgrade deal. For those who are not aware, you can upgrade Standard to Advanced for $795 per CPU. This roughly a 50% discount over the normal upgrade price and is only available until the 15th December this year.

So, as I was putting the benefits case together I was busy drafting out the description of the new fault tolerance feature and selling it as one of the big bonuses. Now I’m very much into the technical aspects of what I do and sometimes get bogged down in the details when trying to make a benefits case. So I went looking for a video which would undoubtley explain it better at management level. So here it is, VMware CTO Steve Herrod explains VMware Fault Tolerance in relatively plain english.

Now for those like me, who do want to know the technical detail, the following link Fault Tolerance 800 x 600 Demo is to a high quality video demonstration of the new Fault Tolerance feature in vSphere.

A word of warning though,   I found a significant caveat on VMware’s Fault Tolerance web page!  It only supports VMs with a single virtual CPU.  This is bad news because I was looking at this as a potential solution to protect an  exchange server which I very much doubt will have a single vCPU.  You can find said Caveat at the bottom of the Fault Tolerance feature page. I’m busy trying to find out from the community at large if there are any timelines on increasing the limit on this.

New Products, VMware, vSphere VMware, VSphere

 Just having a little look through the videos on VMwareTV on YouTube and noticed that they’ve posted a new video on the new vSphere Data Recovery feature, I’ve included it below for your viewing pleasure.

New Products, VMware, vSphere VMware, VSphere