VirtualPro

One of the things I’ve come to love about blogging is the fact that I occasionally get contacted by the odd tech start-up. Keen to demonstrate their latest market leading idea that is going to revolutionise the industry as we know it.  Earlier this month I was contacted by Mindy Anderson who is the Product director at tech start-up Virsto (short for Virtual Storage). Virsto had a new product for Microsoft Hyper-V that they wanted to demonstrate to me in advance of their big product launch. Having looked at Mindy’s background in the storage industry I was very keen to hear more about their new product.

The product is called Virsto One and is aimed solely at Windows 2008 R2 Hyper-V. The product introduces some new features like thin provisioned clones & snapshots, that expand the functionality of the standard Hyper-V product. The most interesting feature in my opinion though is the attempt to tackle the virtualisation / storage problem commonly known as the I/O blender effect.

So what does Virsto One look like?

The software itself installs in the parent partition of each Hyper-V host and consists of the filter driver, a system service and a VSS provider.  The filter driver sits above the raw storage (any block storage) and presents a VHD object to the parent partition.  This setup allows users to configure physical storage once and then use Virsto One to carry out all future provisioning tasks. This includes full support for creating  thin provisioned, high performing, cluster aware snapshots and clones from either the Virsto One Hyper-V MMC snap-in or Powershell.

So what about the I/O blender effect?

Most storage technologies are not designed for the virtual data centre, most are still designed around the one to one physical server to storage model. Think of a number of virtual machines all with predictable I/O behaviour (if you think of them as physical).  What tends to come out of the physical hypervisor host is a large amount of completely random I/O.  Random I/O has an obvious performance impact when compared with sequential I/O so as you increase the number of VM’s you increase the random I/O from your Hyper-V host.  So as VM density increases performance drops, as we all know low VM density is not your objective when you embark on a virtualisation project.

So Virsto One has an interesting way of dealing with this. Although the “secret sauce” has never been divulged in-depth in its basic form they journal the random I/O that comes down from the Hyper-V host to staging disk.  A staging area is required per physical Hyper-V host and about 20GB / 30GB of disk should support multi-terabyte write downs through use of de-dupe technology. Periodically the data in the staging disks will be flushed / written down to the primary storage location, at this point the Random I/O is laid down sequentially on primary storage to improve read performance. Virsto indicated that in time they would look to support multiple de-stages so that data could be de-staged to another array for business continuity purposes or to the cloud for disaster recovery purposes.

Are there any performance figures to back this up?

Performance figures from the Virsto test lab show the I/O Blender effect in full effect as VM density increases in the standard Microsoft setup.  With the Virsto software sitting in the middle, staging the data and de-staging it sequentially, there is an obvious improvement in performance.  These test results were from Virsto’s own lab and I stressed the importance of having these independently benchmarked by customers or an external consultancy.  Wendy indicated to me that this was something they were looking into,  I look forward to reading and sharing the whitepaper when it is eventually produced.

So who would be interested in a product like this?

Well ideally the product would benefit Hyper-V customers who require high density, high performing virtual environments.  Hosting companies making use of Hyper-V for selling virtual server instances may well see Virsto as a good way of increasing performance and reducing costs through the use of golden images, snapshots, etc.  Who knows though,  individual companies with an investment in Hyper-V may well see the benefit in this kind of product.  In a way I see it’s not to dissimilar to a company buying PowerPath/VE to increase I/O performance in a vSphere environment.

It is important to note that although this product has been initially built for Microsoft Hyper-V the principals behind it are hypervisor agnostic.  I asked the question “why Hyper-V?” at the start of my chat with Virsto,  the answer was that Hyper-V had functionality gaps and was easier to integrate into.  VMware on the other hand is a more mature product where VMFS has gone some way to deal with the traditional virtualisation storage problems.  Citrix virtualisation customers will be happy to hear that testing has already begun in the lab with a version of Virsto one for XenServer, ETA unknown at this stage.

So how much does all this cost?

At the time of the interview,  which was a good few weeks back the per socket price being talked about was $1,000 - $2,000 USD per socket, again not to dissimilar to the pricing for EMC PowerPath/VE.

Conclusion?

My impression at the time of the demo and interview was that this was an interesting product and very clever idea. The main selling point for me was the increase in performance, if it can be independently verified you would think the product will simply sell itself.  I look forward to hearing more about Virsto in the future and I am particularly interested to see what they can do for other hypervisors especially VMware vSphere with it’s new storage API’s.

Hyper-V, New Products, Storage hyper-v, Storage, Virsto

I’ve recently been implementing a vSphere deployment and have been looking at the new features introduced as part of Virtual Machine Hardware 7.  Obviously one of the major new components is the new Para Virtualised SCSI (PVSCSI) adapter which I wrote about way back in May 2009.  When it first came out there were a number of posts regarding the much improved I/O Performance and latency reduction this new adapter delivered, such as Chad Sakac’s I/O vSphere performance test post.

So the other day I stumbled across a tweet from Scott Drummond who works in the VMware Performance Engineering team. Following a little reading and a bit of digging around it appears that the use of PVSCSI comes with a small caveat.  It would appear that if you use the PVSCSI adapter with low I/O workloads you can actually get higher latency than you get with the LSI Logic SCSI adapter (see the quote below)

The test results show that PVSCSI is better than LSI Logic, except under one condition–the virtual machine is performing less than 2,000 IOPS and issuing greater than 4 outstanding I/Os.

This particular caveat has come to light following some more in-depth testing of the PVSCSI adapter performance.  The full whitepaper can be found at the following link.

PVSCSI whitepaper - http://www.vmware.com/pdf/vsp_4_pvscsi_perf.pdf

For those who don’t want to read the technical whitepaper, a summary of the issue can be found in the following VMware KB article.

VMware KB 1017652 - http://kb.vmware.com/selfservice/1017652

So basically, as opposed to just using the PVSCSI adapter as default with VMs running version 7 of the virtual hardware have a think about it’s I/O profile and whether the PVSCSI or LSI logic adapter would be best.

Gestalt-IT, VMware, vSphere ESX 4, I/O, PVSCSI, PVSCSI adapter, SCSI, VSphere

In today’s economic climate it’s currently the done thing to sweat existing assets for as long as you possibly can.  At the moment I am working on a vSphere deployment and we are recycling some of our existing ESX 3.5 U4 hosts as part of the project.  So over the weekend I was testing out vMotion between a new host with the Intel Xeon X7460 processor and an old host with the Xeon 7350 processor.  I was getting the following error message displayed which pointed to a feature mismatch relating to the SSE4.1 instruction set.  Thankfully the error pointed me to VMware KB article 1993

Within this KB article it immediately refers you to using Enhanced vMotion Compatibility (EVC) to overcome CPU compatibility issues.  I had never used EVC in anger and wanted to read up on it a bit more before making any further changes.  A quick read of page 190 on the vSphere basic configuration guide gives a very good brief overview for those new to EVC. 

So I was referred to VMware KB article 1003212 which is the main reference for EVC processor support.  Quite quickly I was able to see that EVC was supported for the Intel Xeon 7350 and 7460 using the Intel® Xeon® Core™2 EVC baseline.  In essence as far as vMotion is concerned all processors in the cluster would be equal to an Intel® Xeon® Core™2 (Merom) processor and it’s feature set.  This basically masks the SSE4.1 instruction set on the Intel Xeon 7460 that was causing me the problem.

So I set about enabling my current cluster for EVC,  however when I went to apply the appropriate baseline I was getting the following error displayed. The error related to the host that was currently running 3 Windows 2008 R2 x64 servers.  These servers were obviously using using the advanced features of the Intel Xeon 7460 and as such that host could not be activated for EVC.

The vSphere basic configuration guide (Page 190) makes the following recommendation for rectifying this issue, the example matched my situation exactly.

All virtual machines in the cluster that are running on hosts with a feature set greater than the EVC mode you intend to enable must be powered off or migrated out of the cluster before EVC is enabled. (For example, consider a cluster containing an Intel Xeon Core 2 host and an Intel Xeon 45nm Core 2 host, on which you intend to enable the Intel Xeon Core 2 baseline. The virtual machines on the Intel Xeon Core 2 host can remain powered on, but the virtual machines on the Intel Xeon 45nm Core 2 host must be powered off or migrated out of the cluster.)

Now here is the catch 22, my new vCenter server is virtual and sits on the ESX host giving me the EVC error message.  I had to power it off to configure EVC but I can’t configure the EVC setting on the cluster without vCenter,  how was I going to get round this?  Luckily VMware have another KB Article dealing with exactly this situation.  The aptly titled “Enabling EVC on a cluster when vCenter is running in a virtual machine” was exactly what I was looking for. Although it involved creating a whole new HA / DRS cluster complete with new resource groups, etc it was a lot cheaper than buying a large number of expensive Intel processors. It worked perfectly, rectifying my issue and allowing me to use all servers as intended.

Moral of the story…..… Check out VMware KB article 1003212 for processor compatibility before buying servers and always configure your EVC settings on the cluster before adding any hosts to the cluster.  If it’s to late and you have VMs created already,  well just follow the steps above and you should be fine.

VMware, vCenter, vSphere Cluster, CPU, ESX 4, EVC, Intel, vCenter, vMotion, VSphere

Having put in the first of a number of new vSphere ESX 4 Update 1 hosts a colleague today set about building our new Windows 2008 R2 64 bit vCenter server.  He later informed me that he was not able to boot the Windows 2008 virtual machine using the PVSCSI adapter (Paravirtualised SCSI) 

I was absolutely positive I had read that this was now supported in Update 1. As it turns out it is, however its not quite as straight forward as just adding the PVSCSI adapter and installing windows.  In fact windows will not recognise the boot disk as it has no native driver for the VMware PVSCSI adapter.

There are a couple of ways to get round this,  the first involves creating your VM with a normal SCSI adapter that Windows 2008  does support and then installing Windows.  Once the installation is complete add a second virtual disk with a second controller set up as PVSCSI and then install VMware tools.  VMware tools will deploy the driver required for the PVSCSI adapter,  once installed you can safely reconfigure the original SCSI controller to be PVSCSI and remove the secondary controller and virtual disk.  Now when you reboot your machine you won’t be met with a blue screen of death, instead you will have a fully working Windows 2008 server using all the benefits of the PVSCSI adapter.

For full step by step instructions to complete the above process I recommend using Alan Renouf’s article. For those who prefer to use powershell scripting to make their changes, check out this fantastic script from LucD’s website which will do it all for you.

The above method is one way to get the PVSCSI adapter working on the boot drive but to be honest it’s a bit of a hassle to be doing this every time you deploy a Windows 2008 VM.  So I had a look to see what was involved in obtaining the driver files for loading prior to installation.

First you need to extract the driver files from your vCenter server. You can find the relevant driver files located in the following directory.

C:\Program Files\VMware\VMware Tools\Drivers\pvscsi\

Take the 3 files contained within and make a floppy image, I used UltraISO for this particular task but something like WinImage works just as well. Now you need to boot your VM and once the windows installation files have loaded attach your floppy image.

As you can see in the screenshot below the Windows VM does not pick up the attached virtual disk due to the lack of native driver support.

Once you’ve pointed windows to the floppy image it picks up the VMware PVSCSI controller driver contained upon it.  Click next to apply the driver.

Once the system has applied the driver you can see the virtual disk for your installation.

For those who are looking to add this driver or any other VMware tool driver into a Windows 2008 Pre-Installation environment,  this VMware KB article on how to do it could be handy.

VMware, vSphere Boot Disk, ESX 4, PVSCSI, PVSCSI adapter, VSphere, windows 2008

I’ve just had an email through this morning for the upcoming Scottish VMware User Group.  The next event is due to take place in Glasgow on the afternoon of Thursday the 21st of January.

Interest certainly appears to be picking up, with over 200 registered Scottish VMUG users already. Though there is a long way to go before we reach the dizzy heights set by the Dutch VMUG which has over 3000 members of which 600 attended the most recent VMUG meeting.

If you haven’t registered and are interested in doing so click the link below.

Initial agenda is laid out below,  as always this could be subject to change.

Events Scotland, Scottish VMUG, VMUG, VMware User Group

I’m currently working with my colleagues on an upgrade of our VI 3.5 infrastructure to vSphere Enterprise Plus.  We have recently been mulling over some of the design elements we will have to consider and one of the ones that came up was virtual Distributed Switches (vDS).  We like the look of it,  it saves us having to configure multiple hosts with standard vSwitches and it also has some nice benefits such as enhanced network vMotion support, inbound and outbound traffic shaping and Private VLANs.

One of the questions that struck me was,  what happens if your vCenter server fails? what happens to your networking configuration? Surely your vCenter server couldn’t be a single point of failure for your virtual networking, could it?

Well I did a bit of digging about, chatted to a few people on twitter and the answer is no it would not result in a loss of virtual networking.  In vSphere vDS the switch is split into two distinct elements,  the control plane and the data plane. Previously both elements were host based and configured as such through connection to the host, either directly using the VI client or through vCenter. In vSphere because the control plane and data plane have been separated,  the control plane is now managed using vCenter only and the data plane remains host based.  Hence when your vCenter server fails the data plane is still active as it’s host based where as the control plane is unavailable as it’s vCenter based.

One thing I was not aware of was where all this vDS information is stored . Mike Laverick over at RTFM informed me that the central config for a vDS is stored on shared VMFS within a folder called the .dvsData folder. I’ve since learnt that this location is chosen automatically by vCenter and you can use the net-dvs command to determine that location. It will generally be on shared storage that all ESX hosts participating in the vDS have access to.  As a back up to this .dvsData folder a local database copy is located in /etc/vmware/dvsData.db which I imagine only comes into play if your vCenter server goes down or if your ESX host loses connectivity to the shared VMFS with the .dvsData folder.  You can read more about this over at RTFM

Interesting links if your considering VMware Distributed Switches

VMware’s demo video of vDS in action, for those who want to learn more about vDS

Mike Laverick’s great reasoning on whether you should use vDS or not

Eric Sloof’s vDS caveats and best practices article

VMware’s vSphere Networking white paper explaining new vDS features

VMware’s vSphere Networking white paper on vDS network migrations

Jason Boche’s very interesting article on a specific vCenter + vDS issue

ESX, vCenter vCenter, VSphere

Just before this years VMworld event I wrote about a Project Onyx which was being run by Carter Shanklin over at VMware.  It was in it’s early stages back then and to be honest I haven’t looked back at it for a while, until today.

I got a comment today on my original article from Ben Neise who works over at Dell in Glasgow.  He kindly informed me that the Alpha edition of Project Onyx had been released and can be downloaded by clicking the link.  Project Onyx Alpha Release

Carter has produced a very quick video on the basic usage of the new alpha release. I’ll be taking a closer look at this myself when I get into the office tomorrow, hopefully this is the kick start my VMware CLI scripting needs.

VI Toolkit / Powershell, VMware Project Onyx, VI Toolkit / Powershell

I was doing some work out of hours the other night on my employers Virtual Infrastructure when bang on time the little red triangles started popping up against certain ESX hosts in vCenter.  Why you ask? well it’s AV scanning time on our VM’s of course, or the Sophos summit as we affectionately call it due to its uncanny resemblance to a mountain range when you look at the CPU performance stats in vCenter.

It got me thinking, has any one vendor actually got a product out there utilising the VMSafe API that could help me rid our virtual infrastructure of this problem?

My first stop was of course the main VMSafe page where I did find a large list of official partners who are working on developing products to utilise the VMSafe API. The pleasing thing to see was that there are plenty of mainstream security vendors taking part.  However I’ve still to see any of them releasing a product to market that actually utilises VMSafe.

Earlier this year in Glasgow I heard Mcafee talk about VMSafe as part of the VMware vSphere launch road show.  They talked about building a vApp that could sit in your Virtual Infrastructure and take care of AV scanning with the aim of reducing the CPU overhead that AV scanning introduces. I did a little trawl of the web and couldn’t find anything official, I did however find the following forum post (quoted below) which is definitely the unofficial line.

Virus Scan for Offline images is available, which uses VMSafe APIs to scan offline disks accessed via ESX

Nothing is currently road mapped for on-access scanning - no AV vendor has this technology available (or even road mapped as far as I’m aware) yet.

I did a bit more digging on this “scan offline disks” comment and found a recent article by VMware’s Richard Garsthagen.  This article reveals that a piece of software called the VMware Virtual Disk Development Kit (VDDK) can be used to conduct an offline scan of disks attached to powered on or off virtual machines (quoted below). 

VMware VDDK (also being seen as part of the VMsafe initiative, but has been available for longer). The VDDK is an disk API, that allows other programs to access a virtual machine’s hard disk like the VMware Consolidated Backup solution does. It does not matter is the VM is powered on of off, but a disk can just be ‘extra’ mounted to another virtual machine that for instance runs a virus scanner. The clear downside of VDDK is that nothing is real time.

Surely this would rid me of my daily scheduled Sophos summit, wouldn’t it? Think of a hypothetical scenario where you have a VDI setup with 1000 windows XP VM’s,  imagine the strain put on your ESX clusters by 1000 machines kicking off a scheduled daily AV Scan. Would an appliance that could offline scan disks reduce the strain? Well thinking about it, possibly not.  It would still have to conduct a scan of 1000+ virtual disks, only this time it wouldn’t have nearly as many CPU cycles available to churn through the work. All it would have is the resources assigned to the vApp which is likely to be completely inadequate for such a large task. With this in mind it’s likely that it would probably take a large amount of time to complete.  It could even take longer than a day which wouldn’t be much use for a daily AV scan. I’m sure some companies would rather suffer the ESX CPU resource pain point as opposed to sacrificing security through ineffective or untimely AV Scanning.

Richard’s article along with the solutions tab on the VMSafe webpage did however reveal that a couple of products that use VMSafe have made it to market.  One is called vTrust from Reflex Systems which appears to be a multi faceted application, which according to their site provides dynamic policy enforcement and management, virtual segmentation, virtual quarantine and virtual networking policies.  The other application is a hypervisor based firewall appliance from Altor that supports virtual segmentation and claims to provide better throughput by using the Fast Path element of the VMSafe API.

So it would appear on the surface that progress has been slow.  To only find two VMware certified appliances in the market place was, I have to admit, quite a surprise!  It looks like it’s going to be a while before we see VMsafe being fully utilised by vendors, even then we will  have those wary individuals who will never quite be convinced.

Neil Macdonald of Gartner makes a good point about the potential for VMSafe appliances to introduce possible security vulnerabilities at a lower level in the infrastructure.

If I’m responsible for VM security, I’ll consider it after the APIs ship, after the vendors finally ship their VMSafe-enabled solutions, after I’ve got a level of comfort that these VMSafe-enabled security solutions don’t in of themselves introduce new security vulnerabilities

Edward L Haletky who is very much focused on virtualisation security also makes a good point about low level vulnerabilities and the interaction of multiple VMSafe appliances. 

I fully expect VMware to not only ensure the VMSafe fastpath drivers do nothing harmful to the virtual environment, but also address interaction issues between multiple VMSafe fastpath drivers. In addition, I would like such reports made available to satisfy auditing requirements.

So was VMSafe simply something to bolster the vSphere marketing launch,  an announcement made before it should have been?  Usually VMware are quite good at keeping these kind of things under wraps and releasing them when they are a little more mature and ready for use in real world scenarios.  Now I don’t know what work was done with partners in advance but I would have liked to have seen a couple of the major security vendors releasing appliances at the same time as VMSafe was announced.  For me that certainly would have installed a little more confidence in VMSafe than writing this article has.

If anyone out there is writing appliances utilising the VMSafe API and wants to comment, please do.  I would love to hear some news from the front line as to what is being developed, where it will be applied and when we can expect to see it.

General, Gestalt-IT, vSphere VMsafe, VSphere

Over the last month or so I’ve had two invites to participate in vendor events abroad.  The first was an invite to the Gestalt IT tech day in San Francisco, the second was an invite to the EMC EMEA Customer Council event in Prague.  Now as much as I would love to go to everything I get invited to, I have a day job which pays the bills so in this instance I had to chose the one most relevant to my employer and that was the EMC EMEA Customer Council.

Having never been invited to an EMC Customer Council event before I wasn’t entirely sure what to expect. The basic structure of the event involved EMC sharing product roadmap and strategy, deep diving a few key technologies / strategies and then listening to customer feedback.  The sessions I attended were very interactive round table discussions, with a lot of enterprise customers who were not backward in coming forward with their feelings and opinions. As the sessions went on I started to see why EMC run these events. It would be hard to gain this kind of candid and honest feedback through any other medium, this kind of information is invaluable to a vendor. From my perspective as a customer I got a lot of good insight into roadmap, allowing me to more accurately propose a long term EMC storage strategy for my employer.  I also got to meet and chat to a lot of interesting people and best of all, I got to hear about the experiences of other customers. It was re-assuring to hear that whether you are an SMB IT operation or an enterprise level one, you tend to have very similar issues. The only difference sometimes being the scale of the infrastructure involved.

Now unfortunately unlike the Gestalt IT Tech Field day, the EMC Customer Council is governed by a non-disclosure agreement which means I cannot blog about any of the content discussed. However it’s a small price to pay when you get invited to an extremely well organised, well attended event where all parties involved get something out of it.

It’s easy to see why companies are starting to catch on to the benefits of engaging the customer community directly. In some instances the community becomes a self help group of sorts as well as an alternative marketing channel for a vendor. I often see “a community” leading the way with product information awareness, problem resolution, best practice and procurement advice. The VMware community stands as  one of the best examples of this,  there is a wealth of information out there and it’s not hard to find if you ever need to go looking. In fact if you use twitter or subscribe to an RSS feed like PlanetV12n more often than not the information lands in your lap without you needing to ever look for it.

I wanted to briefly cover off the Gestalt IT tech day. Stephen Foskett the organiser and chief recently set out on a mission to organise a technical field day that vendors would sponsor without the usual NDA’s being in place. Thus allowing the attending bloggers to write about what they saw until they couldn’t possibly write anymore.  He did an exceptional job and I believe the experience didn’t put him off, he’s already looking at organising Gestalt IT Tech Day 2.

Well the attending bloggers wrote post after post and there was lots of good stuff coming out from the vendor visits they participated in. This event is another good example of vendors engaging successfully with the community and everyone getting something out of it. The vendors get a chance to spread the word about their products and services and the bloggers get lots of technical content to put out there for their readers.  Everyone is a winner and that is exactly what a vendor event should be all about.

To read more about the Gestalt IT Tech day and sample some of the many articles written, click the link. What a Tech Field Day!

General, Gestalt-IT, Storage EMC Customer Council, GestaltIT Tech Field Day

I’ve just finished a piece of work upgrading a few hosts to ESX 3.5 Update 4 using the ESX deployment appliance.  As part of the upgrade I install version 8.2.0 of the HP management agents for ESX. Upon completion of the install I realised that I could not see the Qlogic HBA’s within the Insight Manager home page.

Upon investigation it appears that HP have removed the libraries from their software from version 8.0.0 onwards.  Now ideally I would have installed the libraries before installing HP Insight Manager as indicated by Arnim Van Lieshout in his great HP agent upgrade script post.  In my case I had already done the install, so in order to see the HBA’s and the connected HP Storage, I had to carry out the following steps.

I downloaded the relevant Qlogic API library for ESX 3.5 U4 from the following site  
http://support.qlogic.com/support/oem_product_detail_vmware.asp

I created a temporary folder in the /tmp partition called qlogiclib

I unzipped the contents and copied it over to the new folder using WinSCP.

I logged on to the service console and ran the following commands to stop the HP agents.

service hpsmhd stop and service hpasm stop

I then navigated to the /tmp/qlogiclib folder and ran the command ./Install.sh

I then ran the following commands in the order shown below to restart the HP agents.

service hpsmhd start and service hpasm start

To check if it worked I navigated to https://hostname:2381 and checked for the external storage connections entry under the storage section on the home page.

Finally I deleted the /tmp/qlogiclib folder as it was no longer required.

Now the same steps carried out with vSphere ESX 4 does not have the desired effect.  Even though you can use the HBA test script located in the Qlogic API library they will not show up in HP Insight manager (Much to my annoyance).  When I eventually find a solution I will of course provide an update.

ESX, VMware, vSphere esx 3.5, QLogic