I’ve recently been implementing a vSphere deployment and have been looking at the new features introduced as part of Virtual Machine Hardware 7. Obviously one of the major new components is the new Para Virtualised SCSI (PVSCSI) adapter which I wrote about way back in May 2009. When it first came out there were a number of posts regarding the much improved I/O Performance and latency reduction this new adapter delivered, such as Chad Sakac’s I/O vSphere performance test post.
So the other day I stumbled across a tweet from Scott Drummond who works in the VMware Performance Engineering team. Following a little reading and a bit of digging around it appears that the use of PVSCSI comes with a small caveat. It would appear that if you use the PVSCSI adapter with low I/O workloads you can actually get higher latency than you get with the LSI Logic SCSI adapter (see the quote below)
The test results show that PVSCSI is better than LSI Logic, except under one condition–the virtual machine is performing less than 2,000 IOPS and issuing greater than 4 outstanding I/Os.
This particular caveat has come to light following some more in-depth testing of the PVSCSI adapter performance. The full whitepaper can be found at the following link.
So basically, as opposed to just using the PVSCSI adapter as default with VMs running version 7 of the virtual hardware have a think about it’s I/O profile and whether the PVSCSI or LSI logic adapter would be best.
In today’s economic climate it’s currently the done thing to sweat existing assets for as long as you possibly can. At the moment I am working on a vSphere deployment and we are recycling some of our existing ESX 3.5 U4 hosts as part of the project. So over the weekend I was testing out vMotion between a new host with the Intel Xeon X7460 processor and an old host with the Xeon 7350 processor. I was getting the following error message displayed which pointed to a feature mismatch relating to the SSE4.1 instruction set. Thankfully the error pointed me to VMware KB article 1993
Within this KB article it immediately refers you to using Enhanced vMotion Compatibility (EVC) to overcome CPU compatibility issues. I had never used EVC in anger and wanted to read up on it a bit more before making any further changes. A quick read of page 190 on the vSphere basic configuration guide gives a very good brief overview for those new to EVC.
So I was referred to VMware KB article 1003212 which is the main reference for EVC processor support. Quite quickly I was able to see that EVC was supported for the Intel Xeon 7350 and 7460 using the Intel® Xeon® Core™2 EVC baseline. In essence as far as vMotion is concerned all processors in the cluster would be equal to an Intel® Xeon® Core™2 (Merom) processor and it’s feature set. This basically masks the SSE4.1 instruction set on the Intel Xeon 7460 that was causing me the problem.
So I set about enabling my current cluster for EVC, however when I went to apply the appropriate baseline I was getting the following error displayed. The error related to the host that was currently running 3 Windows 2008 R2 x64 servers. These servers were obviously using using the advanced features of the Intel Xeon 7460 and as such that host could not be activated for EVC.
All virtual machines in the cluster that are running on hosts with a feature set greater than the EVC mode you intend to enable must be powered off or migrated out of the cluster before EVC is enabled. (For example, consider a cluster containing an Intel Xeon Core 2 host and an Intel Xeon 45nm Core 2 host, on which you intend to enable the Intel Xeon Core 2 baseline. The virtual machines on the Intel Xeon Core 2 host can remain powered on, but the virtual machines on the Intel Xeon 45nm Core 2 host must be powered off or migrated out of the cluster.)
Now here is the catch 22, my new vCenter server is virtual and sits on the ESX host giving me the EVC error message. I had to power it off to configure EVC but I can’t configure the EVC setting on the cluster without vCenter, how was I going to get round this? Luckily VMware have another KB Article dealing with exactly this situation. The aptly titled “Enabling EVC on a cluster when vCenter is running in a virtual machine” was exactly what I was looking for. Although it involved creating a whole new HA / DRS cluster complete with new resource groups, etc it was a lot cheaper than buying a large number of expensive Intel processors. It worked perfectly, rectifying my issue and allowing me to use all servers as intended.
Moral of the story…..… Check out VMware KB article 1003212 for processor compatibility before buying servers and always configure your EVC settings on the cluster before adding any hosts to the cluster. If it’s to late and you have VMs created already, well just follow the steps above and you should be fine.
Having put in the first of a number of new vSphere ESX 4 Update 1 hosts a colleague today set about building our new Windows 2008 R2 64 bit vCenter server. He later informed me that he was not able to boot the Windows 2008 virtual machine using the PVSCSI adapter (Paravirtualised SCSI)
I was absolutely positive I had read that this was now supported in Update 1. As it turns out it is, however its not quite as straight forward as just adding the PVSCSI adapter and installing windows. In fact windows will not recognise the boot disk as it has no native driver for the VMware PVSCSI adapter.
There are a couple of ways to get round this, the first involves creating your VM with a normal SCSI adapter that Windows 2008 does support and then installing Windows. Once the installation is complete add a second virtual disk with a second controller set up as PVSCSI and then install VMware tools. VMware tools will deploy the driver required for the PVSCSI adapter, once installed you can safely reconfigure the original SCSI controller to be PVSCSI and remove the secondary controller and virtual disk. Now when you reboot your machine you won’t be met with a blue screen of death, instead you will have a fully working Windows 2008 server using all the benefits of the PVSCSI adapter.
For full step by step instructions to complete the above process I recommend using Alan Renouf’s article. For those who prefer to use powershell scripting to make their changes, check out this fantastic script from LucD’s website which will do it all for you.
The above method is one way to get the PVSCSI adapter working on the boot drive but to be honest it’s a bit of a hassle to be doing this every time you deploy a Windows 2008 VM. So I had a look to see what was involved in obtaining the driver files for loading prior to installation.
First you need to extract the driver files from your vCenter server. You can find the relevant driver files located in the following directory.
Take the 3 files contained within and make a floppy image, I used UltraISO for this particular task but something like WinImage works just as well. Now you need to boot your VM and once the windows installation files have loaded attach your floppy image.
As you can see in the screenshot below the Windows VM does not pick up the attached virtual disk due to the lack of native driver support.
Once you’ve pointed windows to the floppy image it picks up the VMware PVSCSI controller driver contained upon it. Click next to apply the driver.
Once the system has applied the driver you can see the virtual disk for your installation.
For those who are looking to add this driver or any other VMware tool driver into a Windows 2008 Pre-Installation environment, this VMware KB article on how to do it could be handy.
I’m currently working with my colleagues on an upgrade of our VI 3.5 infrastructure to vSphere Enterprise Plus. We have recently been mulling over some of the design elements we will have to consider and one of the ones that came up was virtual Distributed Switches (vDS). We like the look of it, it saves us having to configure multiple hosts with standard vSwitches and it also has some nice benefits such as enhanced network vMotion support, inbound and outbound traffic shaping and Private VLANs.
One of the questions that struck me was, what happens if your vCenter server fails? what happens to your networking configuration? Surely your vCenter server couldn’t be a single point of failure for your virtual networking, could it?
Well I did a bit of digging about, chatted to a few people on twitter and the answer is no it would not result in a loss of virtual networking. In vSphere vDS the switch is split into two distinct elements, the control plane and the data plane. Previously both elements were host based and configured as such through connection to the host, either directly using the VI client or through vCenter. In vSphere because the control plane and data plane have been separated, the control plane is now managed using vCenter only and the data plane remains host based. Hence when your vCenter server fails the data plane is still active as it’s host based where as the control plane is unavailable as it’s vCenter based.
One thing I was not aware of was where all this vDS information is stored . Mike Laverick over at RTFM informed me that the central config for a vDS is stored on shared VMFS within a folder called the .dvsData folder. I’ve since learnt that this location is chosen automatically by vCenter and you can use the net-dvs command to determine that location. It will generally be on shared storage that all ESX hosts participating in the vDS have access to. As a back up to this .dvsData folder a local database copy is located in /etc/vmware/dvsData.db which I imagine only comes into play if your vCenter server goes down or if your ESX host loses connectivity to the shared VMFS with the .dvsData folder. You can read more about this over at RTFM
Interesting links if your considering VMware Distributed Switches
I was doing some work out of hours the other night on my employers Virtual Infrastructure when bang on time the little red triangles started popping up against certain ESX hosts in vCenter. Why you ask? well it’s AV scanning time on our VM’s of course, or the Sophos summit as we affectionately call it due to its uncanny resemblance to a mountain range when you look at the CPU performance stats in vCenter.
It got me thinking, has any one vendor actually got a product out there utilising the VMSafe API that could help me rid our virtual infrastructure of this problem?
My first stop was of course the main VMSafe page where I did find a large list of official partners who are working on developing products to utilise the VMSafe API. The pleasing thing to see was that there are plenty of mainstream security vendors taking part. However I’ve still to see any of them releasing a product to market that actually utilises VMSafe.
Earlier this year in Glasgow I heard Mcafee talk about VMSafe as part of the VMware vSphere launch road show. They talked about building a vApp that could sit in your Virtual Infrastructure and take care of AV scanning with the aim of reducing the CPU overhead that AV scanning introduces. I did a little trawl of the web and couldn’t find anything official, I did however find the following forum post (quoted below) which is definitely the unofficial line.
Virus Scan for Offline images is available, which uses VMSafe APIs to scan offline disks accessed via ESX
Nothing is currently road mapped for on-access scanning - no AV vendor has this technology available (or even road mapped as far as I’m aware) yet.
I did a bit more digging on this “scan offline disks” comment and found a recent article by VMware’s Richard Garsthagen. This article reveals that a piece of software called the VMware Virtual Disk Development Kit (VDDK) can be used to conduct an offline scan of disks attached to powered on or off virtual machines (quoted below).
VMware VDDK (also being seen as part of the VMsafe initiative, but has been available for longer). The VDDK is an disk API, that allows other programs to access a virtual machine’s hard disk like the VMware Consolidated Backup solution does. It does not matter is the VM is powered on of off, but a disk can just be ‘extra’ mounted to another virtual machine that for instance runs a virus scanner. The clear downside of VDDK is that nothing is real time.
Surely this would rid me of my daily scheduled Sophos summit, wouldn’t it? Think of a hypothetical scenario where you have a VDI setup with 1000 windows XP VM’s, imagine the strain put on your ESX clusters by 1000 machines kicking off a scheduled daily AV Scan. Would an appliance that could offline scan disks reduce the strain? Well thinking about it, possibly not. It would still have to conduct a scan of 1000+ virtual disks, only this time it wouldn’t have nearly as many CPU cycles available to churn through the work. All it would have is the resources assigned to the vApp which is likely to be completely inadequate for such a large task. With this in mind it’s likely that it would probably take a large amount of time to complete. It could even take longer than a day which wouldn’t be much use for a daily AV scan. I’m sure some companies would rather suffer the ESX CPU resource pain point as opposed to sacrificing security through ineffective or untimely AV Scanning.
Richard’s article along with the solutions tab on the VMSafe webpage did however reveal that a couple of products that use VMSafe have made it to market. One is called vTrust from Reflex Systems which appears to be a multi faceted application, which according to their site provides dynamic policy enforcement and management, virtual segmentation, virtual quarantine and virtual networking policies. The other application is a hypervisor based firewall appliance from Altor that supports virtual segmentation and claims to provide better throughput by using the Fast Path element of the VMSafe API.
So it would appear on the surface that progress has been slow. To only find two VMware certified appliances in the market place was, I have to admit, quite a surprise! It looks like it’s going to be a while before we see VMsafe being fully utilised by vendors, even then we will have those wary individuals who will never quite be convinced.
Neil Macdonald of Gartner makes a good point about the potential for VMSafe appliances to introduce possible security vulnerabilities at a lower level in the infrastructure.
If I’m responsible for VM security, I’ll consider it after the APIs ship, after the vendors finally ship their VMSafe-enabled solutions, after I’ve got a level of comfort that these VMSafe-enabled security solutions don’t in of themselves introduce new security vulnerabilities
Edward L Haletky who is very much focused on virtualisation security also makes a good point about low level vulnerabilities and the interaction of multiple VMSafe appliances.
I fully expect VMware to not only ensure the VMSafe fastpath drivers do nothing harmful to the virtual environment, but also address interaction issues between multiple VMSafe fastpath drivers. In addition, I would like such reports made available to satisfy auditing requirements.
So was VMSafe simply something to bolster the vSphere marketing launch, an announcement made before it should have been? Usually VMware are quite good at keeping these kind of things under wraps and releasing them when they are a little more mature and ready for use in real world scenarios. Now I don’t know what work was done with partners in advance but I would have liked to have seen a couple of the major security vendors releasing appliances at the same time as VMSafe was announced. For me that certainly would have installed a little more confidence in VMSafe than writing this article has.
If anyone out there is writing appliances utilising the VMSafe API and wants to comment, please do. I would love to hear some news from the front line as to what is being developed, where it will be applied and when we can expect to see it.
An interesting problem occurred the other day with one of our older production ESX 3.0.2 hosts. For the first time with any ESX host we have the service console memory ran out, this resulted in all VM’s becoming unresponsive and loss of service to our users.
Now these hosts were built a couple of years ago by a consultant and all had their service console memory set to the default value of 272MB. I’m in the process of upgrading all hosts to ESX 3.5 U4 and changing the memory levels to the maximum 800MB, this particular host was due to be upgraded in the next 2 weeks. Unfortunate timing!!
VMware support were as helpful as ever and informed my colleague to up the service console memory to 800MB. My only concern was the fact that your swap space is meant to be twice your service console memory. If the memory was only set to 272MB you can be sure that the swap partition wasn’t going to be set to 1600MB.
My colleague was having trouble finding out what size the swap partition was so I gave him a hand. First of all he was doing a df –k at the service console, which shows him the named linux partitions but not the swap partition we were looking for. To get information on all disks and partitions attached to the host we need to run fdisk – l
This command showed us the swap partition created was made up of 1044225 blocks, though we weren’t sure exactly what this equated to in MB.
I took a look at one of our newly built ESX 3.5 U4 hosts and compared it’s fdisk –l results to the scripts used to build it. I quickly found that by dividing by 1024 you could get the size of the partitions. So in this case the swap partition on the ESX 3.0.2 host was roughly 1GB which was less than the recommended 2 x console memory sizing.
On this occasion VMware support advised us that it should be OK as it was. That coupled with the fact we are going to rebuild the server in the coming weeks was enough for us to call the case closed.
However what if we did want to change it? I’d always been taught that changing the swap partition after the host had been built usually meant a full rebuild. However as I’ve been working my way through Scott Lowe’sMastering Vmware vSphere 4 book I came across the steps to do it without a rebuild. It’s always recommended to rebuild a host as opposed to take this action, however occasionally needs must.
first create a new swap file on an existing service console partition, the command below will create a 1.6GB within the path entered /path/to/
Use the following command to turn this into a usable swap file
mkswap /path/to/swap.file
Now enable the swap file with the following command
swapon /path/to/swap.file
If you do try this, it is entirely at your own risk. I haven’t as I am planning to rebuild in the near future. If I wasn’t I would probably have given this a shot just to put my mind at ease.
I recently had a number of vSphere ESX4i USB Key installs following my article on putting vSphere ESX4i on a USB key / Pen Drive. I needed to format a couple for general windows usage, only to find that the ESX4i image creates a number of partitions on the USB Key. Unfortunately Windows does not appear to support the removal of partitions on removable devices so when I was trying to format a 2GB USB stick I was able to format a 110MB partition and that was it. I was a bit stuck on the best way to rectify the issue and wasn’t finding much to help out on the web.
That’s when I stumbled upon the HP USB Storage Format Tool, a great little tool that works with a wide range of USB sticks and not just HP ones. It allowed me to wipe the USB key as a single entity and didn’t care about the partitioning, returning my USB Key to a useable state within windows.
I’ve recently been evaluating some of the new features in VMware vSphere to see what use they would be to my current employer. One of the areas that I touched upon in my “what’s new in vSphere Storage” blog post was thin provisioning. I wanted to come back and cover this particular topic in more detail as it’s a key feature and it’s available throughout all versions of vSphere so I’m sure everyone will be interested in it.
What is Thin Provisioning?
Thin Provisioning in it’s simplest form is only using the disk space you need. Traditionally with virtual machines if you create a 500GB virtual disk it will use 500GB of your VMFS datastore. With Thin Provisioning you can create a 500GB virtual disk, but if only 100GB is in use only 100GB of your VMFS datastore will be utilised. Credit to Chad Sakac for the diagrams below.
How does it work in vSphere?
Thin Provisioning is being heralded as something new with vSphere, when in truth it was already available in VI3. In VI3 creating a thin provisioned disk involved using vmkfstools and was also not a production supported VM configuration. Now in vSphere the creation of thin provisioned disks can be carried out from the VI Client (see below) and is a supported production configuration for a VM.
It’s as simple as checking a check box, the results are pretty good to. Below you can see I have created two thin provisioned VM’s on my new ESX4i host and you can see the provisioned space and the used space stats being shown in the VI Client.
What are the benefits?
The thin provisioning feature is perfect for my home lab environment where disk space is at a premium, but how does it translate into real world implementations of ESX. Well I for one have been looking at exactly this to identify what benefits could be achieved within my employers ESX estate. A quick audit found that our development and system test ESX environment was running at 48% disk utilisation, so straight away thin provisioning would save us 52% on storage capacity used. Paul Manning of VMware mentioned on a recent communities podcast that on average vSphere would save users 50% on storage. This is possibly not such a big thing when your talking about test environments, but when you move up to production SAN Storage, saving 50% on an expensive SAN array is a very real and tangabile cost saving. One that people should definately take into account when making a cost benefit case for buying or upgrading to vSphere.
What are the potential downsides?
One of my personal concerns with thin provisioning is the potential overhead on any write activity that would requires the extension of the VMDK file. To me there is an obvious VMFS operation that needs to take place there which would add to the overall time to complete the disk write. When there is a requirement to expand a disk, the VMDK files will increase in increments based on the block size of the underlying VMFS partition, 1MB, 2MB, 4MB or 8MB. So the overhead may be smaller if your VMFS has been formatted with a bigger block size, i.e. for a 16MB write it only has to expand 2 blocks when the VMFS block size is 8MB but would have to expand 16 blocks if it was formatted with the 1MB block size. I can imagine this percieved overhead could put people off using thin provisioned disks for certain production based environments, especially those where there is a lot of write I/O activity, SQL Server or Exchange for example. To counteract that though, the improvements in the VMware I/O Stack should compensate for this performance overhead. This could potentially leave you in a situation where you’ve reduced a VM’s storage footprint and still have performance equal to that experienced in VI3, possibly not a bad trade off. I’d also expect people running their VMware environment on enterprise SAN technologies from the likes of EMC or NetApp to notice minimal performance impact with thin provisioning as SAN memory caches help take up the strain.
Another downside is if you want to use VMware Fault Tolerance to protect a VM then you cannot use thin provisioned disks. To be honest this is a small issue as Fault Tolerance protection is most likely going to be on virtual machines that are important to your organisation. These machines are probably the ones you wouldn’t thin provision in the first place for performance reasons.
Thin provisioning creates it’s own unique problem in that what we’re basically doing here is over provisioning the storage. You need to keep a very close eye on thin provisioning as it’s quite feasabile that your VMFS datastore could fill up and your virtual machines fall over. Not what you want to come into on a Monday morning, or any morning for that matter. So you need to monitor your storage and ensure that there is enough free space. One of the simplest ways to do this is through the use of the new alarms in vSphere that allow you to alert on datastore usage and datastore over provisioning. These should keep you from filling a datastore and killing your VM’s or ESX Servers
One gotcha that you should watch out for is VM swap files, as these are usually stored with your virtual machines vmdk files in the VMFS datastore. In VI3 the swap file was not deleted when a VM was powered down, in vSphere the swap file is deleted on power down and recreated when the VM is powered up. You should be aware of this when over provisioning storage as you could get into a situation whereby you find you can’t power on a VM because there isn’t enough space for the swap file to be created. This becomes more likely as servers and VM configuration maximum’s increase, if you have a VM with 20GB of RAM it’s going to need 20GB of disk space for the swap file. if you have 256GB of RAM in your vSphere host and you allocate it all out to VM’s then you need to think about the 256GB of disk capacity required to service virtual machine swap files.
Storage vMotion
If you’ve already got a VI3 environment then the chances are that your VM’s aren’t thin provisioned, how on earth are you going to take advantage of this new feature? Well if you have purchased a vSphere edition that supports storage vMotion then you can of course migrate the underlying storage and have it thin provisioned during the move. This should allow existing VI3 customers to claim back a lot of space, as I mentioned before I found that our development and test VI environments were only 48% utilised. If I storage vmotion all those VM’s and thin provision at the same time I will free up about 1.5TB of storage that wasn’t being used in the first place.
I’ve included a video below which demonstrates the Storage vMotion and thin provisoning features in vSphere quite nicely, enjoy!
Well fresh from my return from the Citrix iForum I decided to fire head long into installing XenServer in my home lab so I could have a look at it.
I already run VMware vSphere 4i on my home lab which consists of an HP Proliant ML115 G5. Instead of buying another machine to install Xenserver on or rebuilding my current vSphere server I thought I would try and install XenServer inside a virtual machine. As Eric Gray over at vCritical proved you can install vSphere 4 inside a vSphere 4 virtual machine so surely the same would be possible XenServer 5.5, shouldn’t it?
Well the screenshot below should prove exactly that, Xenserver 5.5 successfully running on vSphere 4i
1 - Create a Virtual Machine with custom settings.
2 - Select the new Virtual Machine version 7 hardware.
3 - Select Red Hat Enterprise Linux v5 (64-bit).
4 - 1 vCPU and 1GB of RAM will suffice.
5 - I used the LSI Logic Parallel SCSI Controller.
6 - Create a disk based on your requirements.
7 - Make it thin provisioned if you want, why wouldn’t you?
8 - Connect the ISO image to the VM and start it.
9 - Follow the prompts on screen to complete the install.
I only had one issue during install and that was when the following message appeared, I carried on installing XenServer and it completed without issue.
However when It came to starting up windows based Virtual Machines, like the message above indicated, I couldn’t. XenCenter showed the following error.
Basically because Windows requires the hardware virtualised assist features (Intel VT or AMD-V), hypervisor on top of hypervisor masks this underlying virtualisation assistance and hence Windows can’t operate. What I did manage to get up and running was virtual machines running Debian Lenny 5.0, so at least I had something to play about with and test out XenServer features such as live motion. Linux machines on XenServer start up in a para virtualised mode and are therefore supported where hardware virtualisation assist is not available.
check out the Debian Lenny based DreamLinux desktop edition, this should give you some VM’s to play within your virtualised XenServer environment.
So although I didn’t get XenServer operating like I wanted to in Vmware vSphere, I did get it working enough to play about with it and it’s features. To be honest that’s all I was after in the first place!!
As soon as vSphere was put on General release I downloaded a copy of vSphere 4 ESXi for running on my home lab setup. I’ve only recently started my home lab and the first machine I purchased was the HP Proliant ML115 G5 server. This was following a recommendation by Kiwi Si over at www.techhead.co.uk who has extensively blogged about the suitability of the HP ML110 G5 and ML115 G5 for ESX labs.
If your interested in starting a home lab I can thoroughly recommend these HP server. Simon even has a deal going with ServersPlus.com (UK) for free delivery on either the HP ML110 or ML115 server. Get over to his hot deals page for further details.
Simon also has some great articles on getting ESXi 3.5 running from a USB pen drive. This was a perfect place for me to start as I wanted to take advantage of the ML115 G5’s internal USB port and boot my server from a USB pen drive. So where do you start?
For those of you using an Apple Mac and wanting to conduct this excercise, check out the following article over on Tom Rowan’s blog
What do I need?
A USB Pen Drive that is over 1GB in capacity - nice and cheap at amazon Download the vSphere ESX4i ISO image from here
Download Shareware version of Winrar from here
Download Free Trial version of Winimage from here
How do I do it?
Once you have downloaded the ISO image open it up with WinRAR, make sure you use WinRAR as I had problems with WinZip and UltraISO
Double click on the image.tgz file to open the contents in WinRAR and drill down to the \usr\lib\vmware\installer directory.
Within this folder you will find a file called VMware-VMvisor-big-164009-x86_64.dd.bz2. This is another zip file so double click on it and the contents will be displayed in a seperate WinRAR window.
Once inside extract the file VMware-VMvisor-big-164009-x86_64.dd using WinRAR and copy it somewhere locally on your PC.
Now install and open up the WinImage trial that you downloaded at the start of this process.
Insert your USB key and then select Disk and restore virtual hard disk image on physical drive as per the screenshot below.
Select the physical USB drive from the list and click OK, when prompted for the virtual disk file navigate to the dd file you extracted to your local PC. This will now image your USB Key with the vSphere ESX4i hypervisor.
Once complete stick the USB key in a server / whitebox that supports 64 bit computing and away you go. The screenshot below shows my own HP Proliant ML115 G5 running vSphere ESX 4i and all this from a simple 2GB USB pen drive.
