VirtualPro

I was lucky enough last week to be involved in a Gestalt IT conference call with Symantec.  The conference call was designed to give us all a sneak preview of what Symantec were planning to announce at VMworld 2010 in a couple of weeks.  Unfortunately it was under embargo, that is until today!

There were a couple of announcements being made, Symantec introduced a new NFS storage product called VirtualStore and made some further announcements about NetBackup 7 and new VMware specific features.  However the most interesting announcement on the call for me was the release of Symantec Application HA for VMware.

Symantec have been looking at why customers are not going “the last mile” with virtualisation.  Why are customers not deploying their Tier 1 applications on their virtual platforms? Symantec’s view on this was that customers still have issues with application level failure within guest VM’s.  This product has been designed to fill that void and at present is a product with no real competitors.

As the call progressed the current HA options were described by Symantec and discussed by the group. The obvious one is VMware HA which covers a physical host failure event. Within the VMware HA product there is also VM monitoring which covers you in the event of an OS level failure event, such as a  blue screen.  Then you can of course employ other technologies such as OS level clustering, however you then have to take heed of caveats that hinder the ability to use features such as vMotion and DRS.

I’m always sceptical when I see new virtualisation products, one of my fears is that companies are attempting to just jump on the crest of the wave that is virtualisation. Symantec are obviously a bit more established than your average company, but as always the jury is out until we see a final product doing the business for real.  It transpired during the call that the product is actually based on Symantec Veritas Cluster Server,  a product with a long history in application availability.

Veritas Cluster Server has a lot of in built trigger scenarios for common products such as Microsoft SQL Server, Exchange Server and  IIS.  On top of this built in, out of the box support Symantec also have a VCS development kit allowing for custom scenarios to be written.  I like this approach,  it reminds me of F5 Networks use of the customer community to support the writing of custom rules and features for their product.  If a custom rule or feature has enough demand then they spend the time developing it into their product range.    Perhaps Symantec could look at leveraging their customer base and community in this way and improve the support around VCS trigger scenarios.  One other potential use of the VCS SDK that springs to mind is for application vendors who are making specialist software, CRM, ERP, Finance systems, etc.  They could look to build in Application HA into pre-configured virtual appliances, that would be a great selling point for any software vendor.

The deployment of the product itself takes the form of a guest deployment / agent. Technical deep dive information on the exact integration between the Symantec product and VMware was thin on the ground.  However there was mention of Symantec’s integration with the VMware HA API,  something that I don’t think has been announced by VMware just yet.  The description given to us during the call was that if Symantec Application HA failed to restart the application it could send a downstream API call to VMware HA and ask it to restart the VM’s Operating System.  An interesting concept, something I am sure we’ll hear more about at VMworld.

Licensing for this new product is quite competitive, $350 per virtual machine, a small price to pay for ensuring your Tier 1 application recovery is automated.  Symantec have promised full integration with vCenter Server and the screenshot below shows Symantec Application HA in action monitoring a SQL 2008 server, click on the thumbnail to see a full size image.

If you would like to learn more about Application HA, then get along to VMware and Symantec’s break out session at VMworld. - http://www.vmworld.com/docs/DOC-4658

Alternatively you can listen to a Podcast from Symantec’s Niraj Zaveri discussing the new product.  - http://www.symantec.com/podcasts/detail.jsp?podid=ent_application_ha

General, Gestalt-IT, New Products, VMware HA, Symantec, VMware

I spotted an issue in my vSphere infrastructure this weekend just past. I noticed that one of the main development boxes was showing the dreaded question, redo log out of space, retry or abort?

As it turned out VMware Data Recovery Manager had taken a snapshot as part of it’s back up routine and had failed when trying to remove it.  This coupled with a scheduled SQL maintenance plan caused the delta files for the snapshot to grow to over 250GB in little over 12 hours.

I eventually overcame the issue by adding an extent to the out of space VMFS datastore, this gave me an extra 160GB with which to play the logs back in.  I then used the very handy SnapVMX utility to tell me how much space was required to replay the delta files.  Luckily for me it only required 20 GB as sometimes it can require as much as the size of the original disk. After the snapshot was merged I did a bit of Storage vMotion and reworked the datastore to get rid of the extent (I’m not a fan of using them)

This particular incident was unfortunately unavoidable, it happened at a weekend, was due to VMDR’s failure to remove a snapshot it had created and unfortunately clashed with a disk intensive operation. It did get me thinking though, although I am careful with snapshots and there usage who else in the organisation is not? how do we mitigate this potential risk?

Snapshots are a handy feature, I generally only use them for short periods of time, usually to provide a rollback when patching or changing configurations. Misuse or mismanagement of snapshots can quite quickly lead to problems, something that a recent blog article from VMware Support deals with quite effectively. Entitled ESX Anatomy 101 it’s a must read for anyone trying to gain a good basic understanding of how VMware snapshot work.

I myself have taken a two pronged attack to preventing snapshots causing problems. The first approach is to schedule a very basic PowerShell script that I found on the blog site of Axiom Dynamics.  This simple little script queries your vCenter server for all current snapshots and then sends an email detailing them.  A simple but effective means of keeping an eye on snapshots across the virtual infrastructure.

The second more proactive approach is to use a vCenter alarm at the data centre level to alert when a VM is running from a snapshot. This alarm simply involves emailing a warning when any snapshot is larger than 2GB. This handy video taken from VMware Knowledgebase article 1018029 describes in detail how to set this up, the KB itself also provides step by step instructions.

Alternatives 

There are a number of alternatives available for reporting on snapshots.

Alan Renouf’s Snapshot Reminder – A Powershell script that integrates with AD to send the creator of a snapshot a little reminder when the snapshot is over 2 weeks old.

Alan Renouf’s vCheck Daily Report – Another Powershell script that reports on a large number of areas within the virtual infrastructure.  One of those areas includes snapshots

RVTools – A very handy .Net application by Rob de Veij  that can be used to query your virtual infrastructure for just about everything. You will notice in the screenshot below the vSnapshot tab which should help you identify those rogue snapshots.

In summary,  everyone who works with snapshots should have an understanding of their usage and limitations.  Obviously you can’t always rely on people to do things right, we are only human after all. As a safeguard ensure you have some level of reporting and alerting in place to help you prevent those annoying and time consuming out of space issues occurring.

VI Toolkit / Powershell, VMware powershell, snapshots, vCenter, VMware

Following a recent implementation of VMware Data Recovery manager we ran into a few issues.  We eventually had to kill the virtual appliances due to the issue we were having and as a result we had a couple of virtual machines with outstanding snapshots.  These snapshots were taken by VDR and as a result could not be viewed or deleted using the snapshot manager.

We raised a call with VMware support and they started a WebEx session to look at the issue.  I always love watching VMware support personnel operating at the service console level, I always pick up a command or two that I didn’t know before.  On this occasion the support engineer was using something called SnapVMX to view the hierarchy of snapshots at the virtual disk level.

At first I thought this was an inbuilt VMware command but it turns out it’s not. It was actually a little piece of code that was written by Ruben Garcia.  What does it do?  well the following extract from the download pages explains it pretty well.

• Displays snapshots structure and size of snapshots for every disk on that VM
• Calculates free space needed to commit snapshots for the worst case scenario
• Checks the CID chain of the analysed files and displays a warning if broken.

I’ve included a little demo screenshot to show what it can do. On the left hand side is  a screenshot from Snapshot Manager within vCenter.  On the right hand side is the same VM being viewed with SnapVMX in the service console.  Put the two together and you get a better idea of the snapshot disk hierarchy and the size of each snapshot.

The other interesting feature is that it tells you what space is required to commit the snapshots.  So for example, say you had taken 5 snapshots of a machine as it was being built and configured.  Say that the overall effect of those 5 snapshots is to fill up your VMFS datastore completely. Chances are that you’re not going to be able to commit the snapshots within the current VMFS datastore.  SnapVMX will be able to tell you the worse case scenario on how much space would be required to commit the snapshots.  Armed with this information you could cold migrate to another datastore that has at least that amount of free space in order to allow you to commit the snapshots.  The screenshot below isn’t the best but the best I could do due to the length of the statement.

For the download and full documentation on how to use this piece of code head over to the following web site. Worth a look if you’re a big user of snapshots.

http://geosub.es/vmutils/SnapVMX.Documentation/SnapVMX.Documentation.html

While searching for a link to Ruben Garcia to put on this article I found that he has a blog site and within that I found a link to a superb troubleshooting VM snapshot problems article which I will definitely be keeping a link to and suggest you check out.  Truly excellent stuff Ruben!

General, Gestalt-IT, VMware Datastore, snapshots, VMFS, VMware, VSphere

After a couple of weeks of speculation VMware have today announced their latest VCAP certifications.  VMware have introduced two VMware Certified Advanced Professional exams for holders of the current VCP4 certification. These two new exams cover administration and design and are pre-requisites of the new VCDX4-DCD qualification.  I’ve been waiting for this announcement for a month now as I’ve been keen to work towards VCDX, however wasn’t keen to do it on VI3. To find out more information on the two new certifications as well as the VCDX4 qualification click the links below, or see the updated FAQ page on VMware’s site.

VCAP – Datacenter Administration

The VCAP4-DCA is directed toward System Administrators, Consultants and Technical Support Engineers who can demonstrate their skills in VMware vSphere and VMware vCenter technologies in relation to the datacenter, as well as their knowledge of application and physical-infrastructure services and their integration with the virtual infrastructure.

VCAP – Datacenter Design

The VCAP-DCD is directed toward IT Architects and Consulting Architects who are capable of designing VMware solutions in a multi-site, large enterprise environment.  They have a deep understanding both of VMware core components and their relation to storage and networking, and also of datacenter design methodologies. They also possess knowledge of applications and physical infrastructure, as well as their relationship to the virtual infrastructure

VCDX4 – Datacenter Design

VMware Certified Design Expert (VCDX) is the highest level of VMware certification. This elite group is comprised of design architects highly-skilled in VMware enterprise deployments and the program is designed for veteran professionals who want to validate and demonstrate their expertise in VMware virtual infrastructure.

Certifications, VMware Certification, Certifications, VCAP, VCAP-DCA, VCAP-DCD, VCDX, VCDX4, VCDX4-DCD, VMware

Following VMworld in 2009 a number of articles were written about a tech preview session on IO DRS – Providing performance Isolation to VMs in Shared Storage Environments. I personally thought that this particular technology was a long way off, potentially something we would see in ESX 4.5. However I recently read a couple of articles that indicate it might not be as far away as first thought.

I initially came across an article by VMware’s Scott Drummond in my RSS feeds.  For those that don’t follow Scott, he has his own blog called the Pivot Point which I have found to be a invaluable source of VMware performance related content. The next clue was an article entitled ESX 4.1 feature leak article, I’m sure you can probably guess what the very first feature listed was? It was indeed Storage I/O Control.

Most people will be aware of VMware DRS and it’s usage in measuring and reacting to CPU and Memory contention. In essence SIOC is the same feature but for I/O, utilising I/O latency as the measure and device queue management as the contention control. In the same way as the current DRS feature for memory and CPU, I/O resource allocation will be controlled through the use of share values assigned to the VM.

I hadn’t realised this until now but you can already control share values for VM disk I/O within the setting of a virtual machine (shown above).  The main problem with this is that it is server centric as you can see from the statement below from the VI3 documentation.

Shares is a value that represents the relative metric for controlling disk bandwidth to all virtual machines. The values Low, Normal, High, and Custom are compared to the sum of all shares of all virtual machines on the server and the service console.

Two main problems exist with this current server centric approach.

A) In a cluster, 5 hosts could be accessing VM’s on a single VMFS volume, there may be no contention at host level but lots of contention at VMFS level. This contention would not be controlled by the VM assigned share values.

B) There isn’t a single pane of glass view of how disk shares have been allocated across a host, it appears to only be manageable on a per VM basis.  This makes things a little trickier to manage.

Storage I/O Control (SOIC) deals with the server centric issue by introducing I/O latency monitoring at a VMFS volume level. SOIC reacts when a VMFS volume’s latency crosses a pre-defined level, at this point access to the host queue is throttled based on share value assigned to the VM.  This prevents a single VM getting an unfair share of queue resources at volume level as shown in the before and after diagrams Scott posted in his article.

The solution to the single pane of glass issue is pure speculation on my part. I’d personally be hoping that VMware add a disk tab within the resource allocation views you find on clusters and resource groups.  This would allow you to easily set I/O shares for tiered resource groups, i.e. Production, Test, Development. It would also allow you to further control I/O within the resource groups at a virtual machine level.

Obviously none of the above is a silver bullet! You still need to have a storage system with a fit for purpose design at the backend to service your workloads. It’s also worth remembering that shares introduce another level of complexity into your environment.  If share values are not assigned properly you could of course end up with performance problems caused by the very thing meant to prevent them.

Storage I/O Control (SOIC) looks like a powerful tool for VMware administrators.  I know in my own instance, I have a cluster that is a mix of production and testing workloads.  I have them ring fenced with resource groups for memory and CPU but always have this nagging doubt about HBA queue contention.  This is one of the reasons I wanted to get EMC PowerPath/VE implemented, i.e. use both HBA’s and all available paths to increase the total bandwidth.  Implementing SOIC when it arrives will give me a peace of mind that production workloads will always win out when I/O contention occurs.  I look forward to the possible debut of SOIC in ESX 4.1 when it’s released.

**UPDATE**

Duncan Epping over at Yellow Bricks has located a demo video of SOIC in action.  Although a very basic demonstration,  it gives you an idea of the additional control SOIC will bring.

Gestalt-IT, New Products, VMware DRS, SOIC, vCenter, VMware, VSphere

I stumbled across a tweet the other day by Carter Shanklin who I hope most of you know. For those that don’t, Carter Shanklin  is a product manager over at VMware specialising in the VMware PowerCLI and other automation tools.

His tweet was about a new project called Project Onyx, which on initial inspection is a tool that allows you to see and capture the powershell code behind actions in vSphere vCenter.  This project is at a very early stage as you will probably see in the video embedded below, however this is a very exciting development for those new to PowerCLI.

Some time back I sat my Hyper-V MCTS exam and one of the things I really liked about Microsoft’s SCVMM product was the ability to see the powershell behind the actions you were carrying out.  It was like a head start on powershell automation, giving you a chance to see what was happening, allowing you to dissect, copy it and re-use it however you wanted.

I’m hoping that VMware are planning something similar here,  perhaps a plug-in for vCenter! I personally struggle sometimes with the PowerCLI and the syntax, etc.  If I could see the code behind an action I was trying to automate a good part of the work would be done already.

This project is currently in the early stages of development.  In order to get this project up and running VMware are looking for people to help them Beta test project Onyx.  They have put the call out for a handful of “dedicated people with a burning need for automation”

Get yourself over to VMworld session VM2241 with your business card and put yourself forward to help this promising looking project gain some traction.

VI Toolkit / Powershell, VMware VI Toolkit / Powershell, VMware

Virtual Appliances have been around for quite some time now and it seems with the recent release of vSphere, VMware are looking to take vApps to the next level. The VMware Virtual Appliance Marketplace (VAM) is the one stop shop providing VMware users with a plethora of pre-configured, ready to download appliances.  There are over a 1000 appliances available in the VAM, all of them capable of being deployed in either the private cloud or in a cloud hosted by a 3rd party.  Now the VAM is also offering on-demand trails of certain vApps through selected vCloud partners.  The vCloud vision is slowly taking shape and and although it may be mostly conceptual now, small steps like this are the real building blocks.

With the release of vSphere 4.0 VMware introduced full support for the OVF 1.0 specification.  What does this mean?  well the Open Virtualisation Format is a platform independent, efficient, extensible, and open packaging and distribution format for virtual machines.  It’s virtual platform independent so if vSphere isn’t your platform of choice the vApp will work with Hyper-V and XenServer. OVF virtual machines are optimised for easy distribution are simple to deploy and support single and multi VM configurations.

Now I spend a lot of my time integrating and supporting applications from 3rd party vendors,  some of them mainstream and some of them more specialist. One of the constant issues I have is around how best to deploy them,  what are the best practices, minimum specifications and correct configurations to ensure successful deployment and reliable operation.  This is something I can see vApps helping all of us with, vApps created using the OVF format will allow vendors to build and maintain pre-packaged systems. Pre-configured to be highly compatible, built to the vendors own best practices and best of all built for rapid deployment to customers regardless of virtualised platform or cloud prefernce. The way I look at it,  it’s not to disimilar to Apple’s approach to controlling the hardware and the software they use. The way Apple operate allows them to guarantee better reliability and compatibility because they know what they’re deploying and what they’re deploying it on. A vendor built vApp has the potential to offer the same benefits and simplify the job of vendors, system integrators and application support teams alike.

So how are VMware assisting vendors in their pursuit of vApp bliss? With the creation of VMware Studio of course,  It was a product I’d never heard of until the beta release of VMware Studio 2.0 was announced earlier this month. The latest features are listed below.

Build vApps and virtual appliances (with in-guest OS and application components) compatible with VMware Infrastructure, VMware vSphere 4.0 and the cloud

- Support for OVF 1.0 and 0.9
- Available as an Eclipse plugin in addition to the standalone version
- Ability to accept existing, Studio-created VM builds as input
- Support for 32 bit and 64 bit versions of Windows 2003 and 2008 Server
- Support  for SLES 10.2, RHEL 5.2 and 5.3, CentOS 5.2 and 5.3 and Ubuntu 8.0.4.1
- Publish patches to update deployed virtual appliances
- Extensible in-guest management framework
- Automatic Dependency resolution
- VMware ESX, ESXi, Vmware Server 1.0.4 - 2.0, VMware Workstation enabled as         provisioning engines.
- Infrastructure enhancements in the GUI and builds

So what does a vApp deployment look like,  well here’s a video that was posted on the VMware vApp developer blog.  In this demo a user deploys a mulit VM, multi-tiered version of SugarCRM in just a few clicks with no need to even start up a VM Console. 

New Products, VMware vApps, vCloud, VMware

Well fresh from my return from the Citrix iForum I decided to fire head long into installing XenServer in my home lab so I could have a look at it.

I already run VMware vSphere 4i on my home lab which consists of an HP Proliant ML115 G5.  Instead of buying another machine to install Xenserver on or rebuilding my current vSphere server I thought I would try and install XenServer inside a virtual machine.  As Eric Gray over at vCritical proved you can install vSphere 4 inside a vSphere 4 virtual machine so surely the same would be possible XenServer 5.5, shouldn’t it?

Well the screenshot below should prove exactly that,  Xenserver 5.5 successfully running on vSphere 4i

So how did I conduct the install,  well first of all I downloaded the ISO from Citrix’s website and then did the following.

1 - Create a Virtual Machine with custom settings.
2 - Select the new Virtual Machine version 7 hardware.
3 - Select Red Hat Enterprise Linux v5 (64-bit).
4 - 1 vCPU and 1GB of RAM will suffice.
5 - I used the LSI Logic Parallel SCSI Controller.
6 - Create a disk based between 20 – 30GB (less has caused issues, see comments below)
7 - Make it thin provisioned if you want,  why wouldn’t you?
8 - Connect the ISO image to the VM and start it.
9 - Follow the prompts on screen to complete the install.

I only had one issue during install and that was when the following message appeared,  I carried on installing XenServer and it completed without issue.

However when It came to starting up windows based Virtual Machines,  like the message above indicated, I couldn’t.  XenCenter showed the following error.

Basically because Windows requires the hardware virtualised assist features (Intel VT or AMD-V),  hypervisor on top of hypervisor masks this underlying virtualisation assistance and hence Windows can’t operate.  What I did manage to get up and running was virtual machines running Debian Lenny 5.0,  so at least I had something to play about with and test out XenServer features such as live motion. Linux machines on XenServer start up in a para virtualised mode and are therefore supported where hardware virtualisation assist is not available.

check out the Debian Lenny based DreamLinux desktop edition,  this should give you some VM’s to play within your virtualised XenServer environment.

So although I didn’t get XenServer operating like I wanted to in Vmware vSphere, I did get  it working enough to play about with it and it’s features.  To be honest that’s all I was after in the first place!!

Citrix, VMware, XenServer, vSphere Citrix, VMware, VSphere, XenServer

Following the completion of my post last night about the potential paths that the VMware certification was going to take,  VMware have made an official announcement on the vSphere VCP.

To see the official VMware VCP certification path diagram click the link

For more information check out the following VMware education services page. This confirms that existing VCP’s can sit the exam without sitting the “what’s new in vSphere” course, which is great news !!

Certifications, VMware, vSphere Certifications, VMware, VSphere

I’ve been trying to find out a bit more information on what is going to happen to the VCP qualification now that vSphere has been launched. Way back in January this year I blogged about the impending changes to the VCP exam and since then there have been a number of posts speculating as to what will be involved in retaining or obtaining vSphere VCP status.

The best post by far,  which I admit I only stumbled upon today, is by a gentleman by the name of Arnim van lieshout .  This was originally published back at the beginning of March and refers to vSphere as NGen4 (Next Generation 4). Now, it’s very detailed and official looking,  I’m not sure he did this himself and I have actually seen a few blog posts suggesting that this material is covered by the VMware NDA.  However the cat is out of the bag so I’m going to share it with you anyway,  luckily I’m not covered by any NDA agreement because I’m not a VMware beta tester (though I would like to be one).

What Arnim’s diagram tells us is that there will be a new vSphere Install, Configure and manage course exactly like there was for VI3.  There will also be a what’s new course,  this designed for existing VCP’s who want to upgrade to the new vSphere VCP.  It would also appear that VMWare is keen to maintain the quality of their certification and will continue enforcing the courses as part of the certification path. This is something I am happy to see as it protects my own personal investment in gaining VCP status and makes it mean something.

What’s most interesting is that Arnim’s then describe’s the differences between the coloured lines in the diagram.  Orange lines relate to shortcuts available in the first 9 months from the general release of vSphere and the black lines relate to the mandatory path you must take after 9 months when the shortcuts have been removed. Seeing as it’s been a long day I’ve included out takes from Arnim’s blog post which describe it better than I do.

The options (black arrows):

Everybody new to VMware MUST attend the NGen4 ICM training

Everybody who attended the VI3 training (I&C or DSA only), but didn’t passed the exam MUST  attend the NGen4 ICM training again

If you are a VCP on ESX 2.x you MUST attend the NGen4 training

If you are a VCP on VI3 you MUST attend the NGen4 What’s New training

The shortcut options (orange arrows):

From the date of general release VMware will offer the shortcut paths for 9 months

If you are VCP on VI3 than you are allowed to take the new NGen4 exam WITHOUT attending the NGen4 “What’s New” training

If you are VCP on ESX 2.x you are still allowed to take the VI3 exam in order to become VCP on VI3 and then take the NGen4 exam

If you attended the VI3 training (I&C or DSA only) but didn’t passed the VI3 exam, you are offered 2 options:

Take the VI3 exam, and become VCP on VI3 and then take the NGen4 exam

Attend the NGen4 “What’s New” training and take the NGen4 exam

I wonder how much of this will materialise into hard fact,  we obviously await the official VMware announcements regarding certification.

Another post that I found quite interesting and one that is quite recent was from VMware certified instructor Scott Vessey.  He kindly informs us that once the vSphere launch is done VMware will start beta testing the exam in preperation for wider distribution.  He estimates that Pearson Vue will be offering the exam within 6-8 weeks of the product release.  Personally I will be trying to sit the upgrade exam without the course as my work won’t pay for it this year and I can’t afford it myself. Better get studying hadn’t I !!!!

Certifications, VMware, vSphere Certification, VCP, VMware