VirtualPro

Since last week I have seen a number of questions in and around VASA and how it is configured for EMC arrays.  I got a couple while doing the Q&A for EMC’s recent VNX best practices with vSphere 5 live WebEx and the day after I was asked by Cormac Hogan over at VMware to take a look at a question asked on the VMware blog site.  So I’ll admit now,  I hadn’t really had a chance to look at VASA in-depth, shame on me!  However I thought that this was as good a chance as any to learn and I thought I would do a post on how to configure it for both EMC’s VNX and VMAX systems. Big thank you to my EMC colleague Garrett Hartney for providing both his time and an environment that we could set this up in.

EMC’s VASA implementation

For those not familiar with VASA I strongly suggest reading this article to familiarise yourself with the what and why around this new VMware API. For those who just want the short version, VASA is essentially an API that allows storage vendors to publish array capabilities up to vCenter.  This allows VMware admins to see characteristic information about the storage underpinning their datastores and also allows them to use VMware storage profiles to enforce VM storage placement policy compliance, e.g. This SQL VM will always sit on performance disks.

The table below shows how EMC currently publishes it’s array capabilities through VASA 1.0 up to vCenter.

An example of how this looks for a datastore when pulled through to vCenter can be seen in the screenshot below.

Core Components and architecture

Regardless of which array you are connecting to, EMC’s implementation of VASA is done using Solution Enabler and something known as the SMI-S provider.  Together these two components act as a middle tier between vCenter and the different arrays being queried.  It’s worth pointing out that the SE \ SMI-S server supports in-band (SAN attached) and out of band (network) connectivity for VNX and CLARiiON arrays and in-band connectivity only for Symmetrix arrays. The architecture of the setup is demonstrated in the diagram below.

SE / SMI-S server deployment

To get started with VASA you will need to download SMI-S version 4.3.1 which already comes pre-bundled with Solution Enabler 7.3.1.  This software can be downloaded from the link below and comes with an option for 32 and 64 bit Windows as well as Linux.  For full details on OS support see the release notes for SMI-S 4.3.1 – (PowerLink Account Required for downloads)

Home > Support > Software Downloads and Licensing > Downloads S > SMI-S Provider

As part of my own deployment I am using a Windows 2008 R2 64 bit server to deploy the core components.  The server has been built as standard with no special configuration required.

• We first of all need to deploy Solution Enabler and SMI-S on your designated server, locate the installation media and run the install package.

• When presented with the welcome screen click next.

• Leave the install location as default and click next.

• When prompted select the array provider only and click next.

• Review the installation settings and space requirements and click next to install.

• Once the install is complete, click finish.

• Configure the environment variables on the server to include the SYMCLI path

• Locate the following file and open it for editing

• Locate the line below and change the value from 100 to 1200, save and exit the file.

• Navigate to the services console and restart the ECOM service

• navigate to the location shown below and run testsmiprovider.exe

• the next step is to connect to the SMI-S provider. I used the defaults which are shown in the square brackets, just hit enter on each line to use the default.

• Once connected you will see the following at the command prompt

• type the dv command at the prompt to display version information about the SMI-S provider install.  This basically proves that everything is working as expected

• that concludes the basic installation and configuration of the SMI-S and Solution Enabler server,  now all we need to do is add in the storage arrays we want displayed to vCenter via the VASA api.

CLARiiON and VNX

SUPPORTED - CLARiiON Navisphere Release 28, 29 & 30, VNX Unisphere Release 31
(SMI-S supports many earlier CLARiiON releases but vSphere 5 does not)

Earlier I mentioned that the CLARiiON and VNX arrays could be added to SMI-S in-band or out of band.  The most common method and the one I intend to use here is to connect out of band, i.e. across the network.  If you do want to connect in-band with direct SAN connection then check out page 39 of the SMI-S v4.3.1 release notes.

One major pre-requisite for connecting CLARiiON and VNX is that the user account used to connect to the arrays must be an administrator login with global scope. At this point you should hopefully still be connected to the testsmiprovider.exe application used earlier,  if you are not then please repeat the command line steps shown above to reconnect.

• Once connected successfully type the commands addsys to begin adding the array
• Enter the IP address / DNS name for Storage Processor A and hit enter.
• Enter the IP address / DNS name for Storage Processor B and hit enter.
• You can continue to add additional arrays here or hit enter to move to the next step.
• Accept the default for the address type, i.e. IP/Nodename by hitting enter.
• Continue answering this question for each storage processor / array added
• enter the global scope administration user account for connecting to the arrays.
• enter the password for the administration account being used
• You will then see the message +++++ EMCAddSystem ++++

• After a while you will see the output from the addsys operation, as you can see below the output is 0 which indicates success.  The details of the system added are then listed.

• If you now run the dv command the arrays added will be listed as connected.

• Now that the array is registered we now need to add the VASA provider into vCenter. Log into vCenter and navigate to the home screen, locate and click on the storage providers icon.
  

• Within the storage provider screen click on the add button as shown below.

• Enter a name for the provider and enter the URL shown below,  the IP address of the server hosting SE / SMI-S should be entered where it has been blanked out below.  The user name is admin and the password is #1Password.

• When prompted accept the certificate for the SMI-S provider

• Once successfully added you will see the provider displayed

• Highlight the provider and you will see the array that was connected to the SMI-S provider server earlier.

• To check that VASA is working correctly in vCenter click the VM Storage Profiles icon on the home screen within vCenter.

• When setting up a new Storage Profile you should be able to see the storage capabilities presented to vCenter,  these are shown below and are marked with system.

• Job done, VASA successfully deployed and storage capabilities showing in vCenter!

DMX4, VMAX and VMAXe

SUPPORTED - Enginuity 5875
(SMI-S supports earlier Enginuity releases but vSphere 5 does not)

Now unfortunately I do not have access to a Symmetrix to complete my testing, however the release notes for SMI-S state the following which makes it sound very easy.

When started, the SMI-S Provider automatically discovers all Symmetrix storage arrays connected to the host on which the Array provider is running. No other action is required, such as running a symcfg discover command.

As mentioned earlier Symmetrix discovery is done in-band through small gatekeeper LUNs presented to the SE / SMI-S server.  If it is a virtual server then ensure that the LUNs are presented to the VM as physical mode RDMs.  The SMI-S release notes has the following to say about best practice.

When using the SMI-S Provider to manage Symmetrix arrays, it is recommended that you configure six gatekeepers for each Symmetrix array accessed by the provider. Only set up these gatekeepers for the host on which the SMI-S Provider is running.

So in theory it should be as simple as completing the following steps

• Present the gatekeeper LUNs to the server (physical or virtual)
• Restart the ECOM windows service to restart the SMI-S provider (auto discover arrays)
• Use testsmiprovider.exe tool,  run the dv command, verify Symmetrix array is attached.
• Thanks to my colleague Cody Hosterman (who does have a Symm) for the screenshot.

One point to note, if you have SMI-S installed on the same host as the EMC Control Center (ECC) Symmetrix Agent or Symmetrix Management Console (SMC) there are a couple of steps you need to take to avoid some spurious errors.  Check out page 37 of the SMI-S v4.3.1 release notes for further information on the changes required to avoid this.

Summary

I think the important thing to remember here is that this is version 1.0 of VASA. It may not be the most elegant solution in the world but it is a start on what I think will become a key feature in years to come. We are fast moving into an age where VMs become objects where we simply check a box to ensure our requirement or service level is delivered.  Imagine a scenario where a VM is created and as part of the creation process you select the storage based on the VASA information passed up to vCenter from the array.  Do I want it on a RAID 5 or RAID 6 protected datastore? Do I want it on a RecoverPoint replicated datastore? Do I want it on a vPlex distributed datastore? Do I want it on a datastore that is SRM protected?  Although it is v1.0 you can see the potential use cases for this feature in the future are going to continue to expand.

Some of you may well have seen Chad Sakac’s blog post back in September entitled Help us make VASA (and EMC’s VASA provider) better! It includes a questionnaire with questions about what you, the end customer wants to see from VASA. This is a great chance to have your say and influence how EMC implement VASA going forward, lets make v2.0 of VASA a feature that delivers on the huge potential V1.0 has shown.

EMC, VMware EMC, VASA, VMware, vSphere 5

At EMC the vSpecialist team often end up talking to a lot of customers about EMC’s FREE Virtual Storage Integrator (VSI) Plug-ins for vCenter Server.  Not only do customers love the fact that it is FREE they also love the features delivered. The ability to accurately view, provision and manipulate EMC storage directly within vCenter empowers VI admins and makes everyone’s life that little bit easier.

When I started writing this article we were on version 4.2 of the VSI plug-ins, following VMworld 2011 we are now up to version 5.0 the fifth generation of this excellent VMware / EMC toolkit. The plug-ins that make up the VSI are listed below, to download use the link below or use the cookie trail to navigate to the page on EMC PowerLink.

• VSI Storage Viewer Plug-in 5.0
• VSI unified Storage Management Plug-in 5.0
• VSI Storage Pool Management Plug-in 5.0
• VSI Path Management Plug-in 5.0

Home > Support > Software Downloads and Licensing > Downloads T-Z > Virtual Storage Integrator (VSI) – (Please Note: PowerLink account required)

One of the great features that people are drawn to is the ability to allow VI admins to provision storage directly from within vCenter. This is done with the VSI Unified Plug-in for Celerra, CLARiiON and VNX(e) and done with the VSI Storage Pool Management plug-in for the VMAX. One of the first question I often get asked is how is the secured,  how does the storage team ensure that only the right VMware admins are manipulating the underlying storage?

The answer previously was… well to be honest we didn’t really have an answer to this one. Technically if you allowed the VMware admins to provision storage you needed to trust them not to go provisioning crazy and fill up your storage array.  Obviously that response was not really acceptable for any environment and EMC have been working to rectify that.

The Access Control Utility is a new part of the VSI framework which allows storage administrators to granularly control availability of storage platforms and storage pools on those platforms.  These security profiles when created can be exported and passed to the VMware administrators and imported into the VSI unified storage management plug-in. The following blog post details the steps involved in completing this process for a VNX array in vSphere 4.1

So we start by double clicking on the shiny padlock icon that will have been added to your desktop when you installed the VSI unified storage management plug-in.  When the ACU starts we are presented with the profile management screen.  This will of course be blank the first time you start the utility, in this screenshot below however you can see a couple of existing access profiles I have created for some VNX arrays in the lab.

To Create a new profile you simply click the Add button, you are then presented with the details screen for the new access profile being created.  Here you enter the name of the profile and a suitable description and click next when finished.

The next step in the wizard is where you define the storage system that will be permissioned as part of the security profile.  You click on Add and then select the system you are going to permission,  as you can see the VSI ACU supports Celerra, CLARiiON, VNX and the VNXe arrays. For VMAX you need to look at Storage Pool Manager (SPM) to control access,  I’ll look to blog about this one at a later date.

The next screen presented very much depends on the storage system you select.  If you chose the Celerra option you’re prompted for the details of the control station, username and password.  Select the CLARiiON and you’re prompted for the Storage Processor details and login credentials. If you select the VNXe then you’re promoted for the management IP and the login credentials.  I’m sure you can see the pattern developing here!

In this example we are dealing with a VNX array and as such the option is whether you want to give access to block storage, file storage or both. As both are controlled differently within the VNX, if you select both you will need to enter the IP and credentials for the Storage Processor (Block) and the VNX Control Station.  For the purposes of this example I’m going to use Block only as you can see in the screenshot below.

When you click next you’re prompted to enter the storage processor IP address and log on details as shown below.

Once you are authenticated you get to select the granularity of access you want to provide.  It’s important to note that when the ACU refers to storage pools it means any storage pools and traditional RAID groups that may have been created on the VNX array.  There are 3 options available as you can see in the screenshot below.

• All storage pools
  This option basically gives a VMware Admin free reign to provision LUNs with the VSI all over the array.  A potential use case for this may be a dedicated development VMware environment with its own dedicated array where the storage team don’t care to much about usage.

• No Storage Pools
  This option is a complete lockdown and acts as an explicit deny to prevent any accidental provisioning on an array, i.e. the VSI unified storage management feature cannot talk to the array full stop, it won’t even show up as an option.

• Selected storage pools
  As the name indicates this option allows the selection of certain storage pools for VSI provisioning.  A potential use case here would be a mixed environment where the array is shared between VMware and physical workloads.  As a storage administrator you would grant permission to the VMware storage pools only thus preventing any potential mis-provisioning (not sure that is actually a word but it certainly has its place when we talk about VSI provisioning)

In this example I’ve chosen selected storage pools as I think this is probably the scenario that most people will be looking for the ACU to help them with.  Within the next screen you are presented with a list of all storage pools / RAID groups on the array.  Here you select the storage pools / RAID groups you want to give the VMware admin access to, when your happy with your selection you simply select finish.  Note in the screenshot below that I have select two individual storage pools (one is a RAID group) to be part of this particular storage profile.

Once you’ve completed storage pool selection you are returned to the profile screen,  you can finish your profile creation right here by clicking on finish or you can add additional storage systems if your VMware environment consists of multiple arrays.

Once you have completed the creation of your security profile the next step is to export it so you can pass it over to your VMware admins. To do this simply highlight the Security profile, click on export and save the file

Chose a location to save the file and don’t forget to add a passphrase to the file so that it cannot be misused.

It’s important to remember that the login credentials provided by the storage admin during the ACU profile setup are the ones used when the profile is imported into the VSI.  The VMware admin will see the connection details and username being used but will not see the password. For audit purposes on the array it may be best to setup a dedicated account for use with the VSI and storage profiles. It should also be noted that the full details of the storage profile are encrypted within the profile export file as you can see below.

So now that you’ve finished creating your storage profile you can pass it on to the VMware administrators to import into the VSI.  To do this you go into vCenter and open up the EMC VSI screen from the home screen.  Click on the Unified Storage Management feature,  then click on add and select Import Access Profile before clicking next.

You now select the XML file created by exporting the ACU storage profile, you enter the passphrase you selected and click next.

As you can see below the VNX array has been added to the VSI and provisioning access is marked as Restricted.  This is as expected as we configured the profile to give access to only two storage pools, FAST_Pool_3_Tier and RAID Group 10.

When you use the EMC VSI to provision storage you will be presented with the VNX array that was part of the imported profile.  You select the storage array and as you can see in the screenshot below you can only create storage on the two storage pools that were added to the ACU storage profile.

Summary

The EMC Access Control Utility was something I have been looking to write about for a while. Since it’s release I’ve often wondered how exactly it worked, what it could / could not do and how it could better meet customer needs. The steps above show that it is possible for a storage team to delegate control of storage pools so VMware admins can quickly provision the storage that they need. Becoming more efficient is something we as vSpecialists talk about on a daily basis, this tool is one of those first steps that you can take to make life easier.  If you are a VMware admin who is working with EMC storage then I suggest you speak to your storage team about this.  Likewise if you are a storage admin, reach out to your VMware counterparts and discuss how this could save you both time in the long term.

Video

My boss Chad Sakacc put a video together for VMworld 2011 which maybe explains it better (certainly quicker) than I maybe have in this blog post. I left it to the end though so you read the article before discovering it . My step by step approach is simply so I can fully understand how it fits together and as I go deal with the many “what if” or “how does that work” kind of questions.  Hope you find it useful in some way, feel free to comment or ask questions.

EMC, New Products, VMware ACU, EMC, VMware, VSI

Some may have picked up on this recently and others may not have even noticed but at the start of this week I started working for EMC.  As of last Monday I am now a technical vSpecialist in Chad’s army covering the UK north region as part of the wider EMEA team of vSpecialists.

So let me start off by saying this is a a huge move for me, one I am so excited about and it is a fantastic opportunity that I fully intend to grab with both hands.  I was previously working on the customer side of the fence and although I enjoyed it immensely and it taught me a lot, the lack of bleeding / cutting edge exposure often left me a little underwhelmed.  To put that in context,  I worked in the investment industry and risk awareness and being risk averse is part and parcel of that industry.  As a result bleeding / cutting edge technologies are eyed with suspicion and only implemented when fully proven or half the industry has implemented it. It always made sense to me from my previous employers perspective but it wasn’t fulfilling my desire to work with the latest and greatest kit.

This is where EMC and more specifically the vSpecialist team comes in! I now find myself embarking on my own personal journey, moving from the buy side to the sell side which I’m sure will be an eye opener. I’m looking forward to working hard to learn all about EMC and it’s plethora of solutions while also working to support and promote the “journey to the cloud” for EMC customers both new and existing.

What does this mean for the VirtualPro blog? Well of course I fully intend to keep it going and hope to have more to write about in the future now that my focus is primarily technology based.  Although I will no longer be viewed as independent I am a technology lover first and foremost.  If it’s worth writing about I will write about it, after all this blog is my own and has been for a number of years, it is not something EMC asks me to do.  I look forward to sharing some new and exciting content with you as my time in the industry progresses.

General EMC, VMware, vSpecialist

I was lucky enough last week to be involved in a Gestalt IT conference call with Symantec.  The conference call was designed to give us all a sneak preview of what Symantec were planning to announce at VMworld 2010 in a couple of weeks.  Unfortunately it was under embargo, that is until today!

There were a couple of announcements being made, Symantec introduced a new NFS storage product called VirtualStore and made some further announcements about NetBackup 7 and new VMware specific features.  However the most interesting announcement on the call for me was the release of Symantec Application HA for VMware.

Symantec have been looking at why customers are not going “the last mile” with virtualisation.  Why are customers not deploying their Tier 1 applications on their virtual platforms? Symantec’s view on this was that customers still have issues with application level failure within guest VM’s.  This product has been designed to fill that void and at present is a product with no real competitors.

As the call progressed the current HA options were described by Symantec and discussed by the group. The obvious one is VMware HA which covers a physical host failure event. Within the VMware HA product there is also VM monitoring which covers you in the event of an OS level failure event, such as a  blue screen.  Then you can of course employ other technologies such as OS level clustering, however you then have to take heed of caveats that hinder the ability to use features such as vMotion and DRS.

I’m always sceptical when I see new virtualisation products, one of my fears is that companies are attempting to just jump on the crest of the wave that is virtualisation. Symantec are obviously a bit more established than your average company, but as always the jury is out until we see a final product doing the business for real.  It transpired during the call that the product is actually based on Symantec Veritas Cluster Server,  a product with a long history in application availability.

Veritas Cluster Server has a lot of in built trigger scenarios for common products such as Microsoft SQL Server, Exchange Server and  IIS.  On top of this built in, out of the box support Symantec also have a VCS development kit allowing for custom scenarios to be written.  I like this approach,  it reminds me of F5 Networks use of the customer community to support the writing of custom rules and features for their product.  If a custom rule or feature has enough demand then they spend the time developing it into their product range.    Perhaps Symantec could look at leveraging their customer base and community in this way and improve the support around VCS trigger scenarios.  One other potential use of the VCS SDK that springs to mind is for application vendors who are making specialist software, CRM, ERP, Finance systems, etc.  They could look to build in Application HA into pre-configured virtual appliances, that would be a great selling point for any software vendor.

The deployment of the product itself takes the form of a guest deployment / agent. Technical deep dive information on the exact integration between the Symantec product and VMware was thin on the ground.  However there was mention of Symantec’s integration with the VMware HA API,  something that I don’t think has been announced by VMware just yet.  The description given to us during the call was that if Symantec Application HA failed to restart the application it could send a downstream API call to VMware HA and ask it to restart the VM’s Operating System.  An interesting concept, something I am sure we’ll hear more about at VMworld.

Licensing for this new product is quite competitive, $350 per virtual machine, a small price to pay for ensuring your Tier 1 application recovery is automated.  Symantec have promised full integration with vCenter Server and the screenshot below shows Symantec Application HA in action monitoring a SQL 2008 server, click on the thumbnail to see a full size image.

If you would like to learn more about Application HA, then get along to VMware and Symantec’s break out session at VMworld. - http://www.vmworld.com/docs/DOC-4658

Alternatively you can listen to a Podcast from Symantec’s Niraj Zaveri discussing the new product.  - http://www.symantec.com/podcasts/detail.jsp?podid=ent_application_ha

General, Gestalt-IT, New Products, VMware HA, Symantec, VMware

I spotted an issue in my vSphere infrastructure this weekend just past. I noticed that one of the main development boxes was showing the dreaded question, redo log out of space, retry or abort?

As it turned out VMware Data Recovery Manager had taken a snapshot as part of it’s back up routine and had failed when trying to remove it.  This coupled with a scheduled SQL maintenance plan caused the delta files for the snapshot to grow to over 250GB in little over 12 hours.

I eventually overcame the issue by adding an extent to the out of space VMFS datastore, this gave me an extra 160GB with which to play the logs back in.  I then used the very handy SnapVMX utility to tell me how much space was required to replay the delta files.  Luckily for me it only required 20 GB as sometimes it can require as much as the size of the original disk. After the snapshot was merged I did a bit of Storage vMotion and reworked the datastore to get rid of the extent (I’m not a fan of using them)

This particular incident was unfortunately unavoidable, it happened at a weekend, was due to VMDR’s failure to remove a snapshot it had created and unfortunately clashed with a disk intensive operation. It did get me thinking though, although I am careful with snapshots and there usage who else in the organisation is not? how do we mitigate this potential risk?

Snapshots are a handy feature, I generally only use them for short periods of time, usually to provide a rollback when patching or changing configurations. Misuse or mismanagement of snapshots can quite quickly lead to problems, something that a recent blog article from VMware Support deals with quite effectively. Entitled ESX Anatomy 101 it’s a must read for anyone trying to gain a good basic understanding of how VMware snapshot work.

I myself have taken a two pronged attack to preventing snapshots causing problems. The first approach is to schedule a very basic PowerShell script that I found on the blog site of Axiom Dynamics.  This simple little script queries your vCenter server for all current snapshots and then sends an email detailing them.  A simple but effective means of keeping an eye on snapshots across the virtual infrastructure.

The second more proactive approach is to use a vCenter alarm at the data centre level to alert when a VM is running from a snapshot. This alarm simply involves emailing a warning when any snapshot is larger than 2GB. This handy video taken from VMware Knowledgebase article 1018029 describes in detail how to set this up, the KB itself also provides step by step instructions.

Alternatives 

There are a number of alternatives available for reporting on snapshots.

Alan Renouf’s Snapshot Reminder – A Powershell script that integrates with AD to send the creator of a snapshot a little reminder when the snapshot is over 2 weeks old.

Alan Renouf’s vCheck Daily Report – Another Powershell script that reports on a large number of areas within the virtual infrastructure.  One of those areas includes snapshots

RVTools – A very handy .Net application by Rob de Veij  that can be used to query your virtual infrastructure for just about everything. You will notice in the screenshot below the vSnapshot tab which should help you identify those rogue snapshots.

In summary,  everyone who works with snapshots should have an understanding of their usage and limitations.  Obviously you can’t always rely on people to do things right, we are only human after all. As a safeguard ensure you have some level of reporting and alerting in place to help you prevent those annoying and time consuming out of space issues occurring.

VI Toolkit / Powershell, VMware powershell, snapshots, vCenter, VMware

Following a recent implementation of VMware Data Recovery manager we ran into a few issues.  We eventually had to kill the virtual appliances due to the issue we were having and as a result we had a couple of virtual machines with outstanding snapshots.  These snapshots were taken by VDR and as a result could not be viewed or deleted using the snapshot manager.

We raised a call with VMware support and they started a WebEx session to look at the issue.  I always love watching VMware support personnel operating at the service console level, I always pick up a command or two that I didn’t know before.  On this occasion the support engineer was using something called SnapVMX to view the hierarchy of snapshots at the virtual disk level.

At first I thought this was an inbuilt VMware command but it turns out it’s not. It was actually a little piece of code that was written by Ruben Garcia.  What does it do?  well the following extract from the download pages explains it pretty well.

• Displays snapshots structure and size of snapshots for every disk on that VM
• Calculates free space needed to commit snapshots for the worst case scenario
• Checks the CID chain of the analysed files and displays a warning if broken.

I’ve included a little demo screenshot to show what it can do. On the left hand side is  a screenshot from Snapshot Manager within vCenter.  On the right hand side is the same VM being viewed with SnapVMX in the service console.  Put the two together and you get a better idea of the snapshot disk hierarchy and the size of each snapshot.

The other interesting feature is that it tells you what space is required to commit the snapshots.  So for example, say you had taken 5 snapshots of a machine as it was being built and configured.  Say that the overall effect of those 5 snapshots is to fill up your VMFS datastore completely. Chances are that you’re not going to be able to commit the snapshots within the current VMFS datastore.  SnapVMX will be able to tell you the worse case scenario on how much space would be required to commit the snapshots.  Armed with this information you could cold migrate to another datastore that has at least that amount of free space in order to allow you to commit the snapshots.  The screenshot below isn’t the best but the best I could do due to the length of the statement.

For the download and full documentation on how to use this piece of code head over to the following web site. Worth a look if you’re a big user of snapshots.

http://geosub.es/vmutils/SnapVMX.Documentation/SnapVMX.Documentation.html

While searching for a link to Ruben Garcia to put on this article I found that he has a blog site and within that I found a link to a superb troubleshooting VM snapshot problems article which I will definitely be keeping a link to and suggest you check out.  Truly excellent stuff Ruben!

General, Gestalt-IT, VMware Datastore, snapshots, VMFS, VMware, VSphere

After a couple of weeks of speculation VMware have today announced their latest VCAP certifications.  VMware have introduced two VMware Certified Advanced Professional exams for holders of the current VCP4 certification. These two new exams cover administration and design and are pre-requisites of the new VCDX4-DCD qualification.  I’ve been waiting for this announcement for a month now as I’ve been keen to work towards VCDX, however wasn’t keen to do it on VI3. To find out more information on the two new certifications as well as the VCDX4 qualification click the links below, or see the updated FAQ page on VMware’s site.

VCAP – Datacenter Administration

The VCAP4-DCA is directed toward System Administrators, Consultants and Technical Support Engineers who can demonstrate their skills in VMware vSphere and VMware vCenter technologies in relation to the datacenter, as well as their knowledge of application and physical-infrastructure services and their integration with the virtual infrastructure.

VCAP – Datacenter Design

The VCAP-DCD is directed toward IT Architects and Consulting Architects who are capable of designing VMware solutions in a multi-site, large enterprise environment.  They have a deep understanding both of VMware core components and their relation to storage and networking, and also of datacenter design methodologies. They also possess knowledge of applications and physical infrastructure, as well as their relationship to the virtual infrastructure

VCDX4 – Datacenter Design

VMware Certified Design Expert (VCDX) is the highest level of VMware certification. This elite group is comprised of design architects highly-skilled in VMware enterprise deployments and the program is designed for veteran professionals who want to validate and demonstrate their expertise in VMware virtual infrastructure.

Certifications, VMware Certification, Certifications, VCAP, VCAP-DCA, VCAP-DCD, VCDX, VCDX4, VCDX4-DCD, VMware

Following VMworld in 2009 a number of articles were written about a tech preview session on IO DRS – Providing performance Isolation to VMs in Shared Storage Environments. I personally thought that this particular technology was a long way off, potentially something we would see in ESX 4.5. However I recently read a couple of articles that indicate it might not be as far away as first thought.

I initially came across an article by VMware’s Scott Drummond in my RSS feeds.  For those that don’t follow Scott, he has his own blog called the Pivot Point which I have found to be a invaluable source of VMware performance related content. The next clue was an article entitled ESX 4.1 feature leak article, I’m sure you can probably guess what the very first feature listed was? It was indeed Storage I/O Control.

Most people will be aware of VMware DRS and it’s usage in measuring and reacting to CPU and Memory contention. In essence SIOC is the same feature but for I/O, utilising I/O latency as the measure and device queue management as the contention control. In the same way as the current DRS feature for memory and CPU, I/O resource allocation will be controlled through the use of share values assigned to the VM.

I hadn’t realised this until now but you can already control share values for VM disk I/O within the setting of a virtual machine (shown above).  The main problem with this is that it is server centric as you can see from the statement below from the VI3 documentation.

Shares is a value that represents the relative metric for controlling disk bandwidth to all virtual machines. The values Low, Normal, High, and Custom are compared to the sum of all shares of all virtual machines on the server and the service console.

Two main problems exist with this current server centric approach.

A) In a cluster, 5 hosts could be accessing VM’s on a single VMFS volume, there may be no contention at host level but lots of contention at VMFS level. This contention would not be controlled by the VM assigned share values.

B) There isn’t a single pane of glass view of how disk shares have been allocated across a host, it appears to only be manageable on a per VM basis.  This makes things a little trickier to manage.

Storage I/O Control (SOIC) deals with the server centric issue by introducing I/O latency monitoring at a VMFS volume level. SOIC reacts when a VMFS volume’s latency crosses a pre-defined level, at this point access to the host queue is throttled based on share value assigned to the VM.  This prevents a single VM getting an unfair share of queue resources at volume level as shown in the before and after diagrams Scott posted in his article.

The solution to the single pane of glass issue is pure speculation on my part. I’d personally be hoping that VMware add a disk tab within the resource allocation views you find on clusters and resource groups.  This would allow you to easily set I/O shares for tiered resource groups, i.e. Production, Test, Development. It would also allow you to further control I/O within the resource groups at a virtual machine level.

Obviously none of the above is a silver bullet! You still need to have a storage system with a fit for purpose design at the backend to service your workloads. It’s also worth remembering that shares introduce another level of complexity into your environment.  If share values are not assigned properly you could of course end up with performance problems caused by the very thing meant to prevent them.

Storage I/O Control (SOIC) looks like a powerful tool for VMware administrators.  I know in my own instance, I have a cluster that is a mix of production and testing workloads.  I have them ring fenced with resource groups for memory and CPU but always have this nagging doubt about HBA queue contention.  This is one of the reasons I wanted to get EMC PowerPath/VE implemented, i.e. use both HBA’s and all available paths to increase the total bandwidth.  Implementing SOIC when it arrives will give me a peace of mind that production workloads will always win out when I/O contention occurs.  I look forward to the possible debut of SOIC in ESX 4.1 when it’s released.

**UPDATE**

Duncan Epping over at Yellow Bricks has located a demo video of SOIC in action.  Although a very basic demonstration,  it gives you an idea of the additional control SOIC will bring.

Gestalt-IT, New Products, VMware DRS, SOIC, vCenter, VMware, VSphere

I stumbled across a tweet the other day by Carter Shanklin who I hope most of you know. For those that don’t, Carter Shanklin  is a product manager over at VMware specialising in the VMware PowerCLI and other automation tools.

His tweet was about a new project called Project Onyx, which on initial inspection is a tool that allows you to see and capture the powershell code behind actions in vSphere vCenter.  This project is at a very early stage as you will probably see in the video embedded below, however this is a very exciting development for those new to PowerCLI.

Some time back I sat my Hyper-V MCTS exam and one of the things I really liked about Microsoft’s SCVMM product was the ability to see the powershell behind the actions you were carrying out.  It was like a head start on powershell automation, giving you a chance to see what was happening, allowing you to dissect, copy it and re-use it however you wanted.

I’m hoping that VMware are planning something similar here,  perhaps a plug-in for vCenter! I personally struggle sometimes with the PowerCLI and the syntax, etc.  If I could see the code behind an action I was trying to automate a good part of the work would be done already.

This project is currently in the early stages of development.  In order to get this project up and running VMware are looking for people to help them Beta test project Onyx.  They have put the call out for a handful of “dedicated people with a burning need for automation”

Get yourself over to VMworld session VM2241 with your business card and put yourself forward to help this promising looking project gain some traction.

VI Toolkit / Powershell, VMware VI Toolkit / Powershell, VMware

Virtual Appliances have been around for quite some time now and it seems with the recent release of vSphere, VMware are looking to take vApps to the next level. The VMware Virtual Appliance Marketplace (VAM) is the one stop shop providing VMware users with a plethora of pre-configured, ready to download appliances.  There are over a 1000 appliances available in the VAM, all of them capable of being deployed in either the private cloud or in a cloud hosted by a 3rd party.  Now the VAM is also offering on-demand trails of certain vApps through selected vCloud partners.  The vCloud vision is slowly taking shape and and although it may be mostly conceptual now, small steps like this are the real building blocks.

With the release of vSphere 4.0 VMware introduced full support for the OVF 1.0 specification.  What does this mean?  well the Open Virtualisation Format is a platform independent, efficient, extensible, and open packaging and distribution format for virtual machines.  It’s virtual platform independent so if vSphere isn’t your platform of choice the vApp will work with Hyper-V and XenServer. OVF virtual machines are optimised for easy distribution are simple to deploy and support single and multi VM configurations.

Now I spend a lot of my time integrating and supporting applications from 3rd party vendors,  some of them mainstream and some of them more specialist. One of the constant issues I have is around how best to deploy them,  what are the best practices, minimum specifications and correct configurations to ensure successful deployment and reliable operation.  This is something I can see vApps helping all of us with, vApps created using the OVF format will allow vendors to build and maintain pre-packaged systems. Pre-configured to be highly compatible, built to the vendors own best practices and best of all built for rapid deployment to customers regardless of virtualised platform or cloud prefernce. The way I look at it,  it’s not to disimilar to Apple’s approach to controlling the hardware and the software they use. The way Apple operate allows them to guarantee better reliability and compatibility because they know what they’re deploying and what they’re deploying it on. A vendor built vApp has the potential to offer the same benefits and simplify the job of vendors, system integrators and application support teams alike.

So how are VMware assisting vendors in their pursuit of vApp bliss? With the creation of VMware Studio of course,  It was a product I’d never heard of until the beta release of VMware Studio 2.0 was announced earlier this month. The latest features are listed below.

Build vApps and virtual appliances (with in-guest OS and application components) compatible with VMware Infrastructure, VMware vSphere 4.0 and the cloud

- Support for OVF 1.0 and 0.9
- Available as an Eclipse plugin in addition to the standalone version
- Ability to accept existing, Studio-created VM builds as input
- Support for 32 bit and 64 bit versions of Windows 2003 and 2008 Server
- Support  for SLES 10.2, RHEL 5.2 and 5.3, CentOS 5.2 and 5.3 and Ubuntu 8.0.4.1
- Publish patches to update deployed virtual appliances
- Extensible in-guest management framework
- Automatic Dependency resolution
- VMware ESX, ESXi, Vmware Server 1.0.4 - 2.0, VMware Workstation enabled as         provisioning engines.
- Infrastructure enhancements in the GUI and builds

So what does a vApp deployment look like,  well here’s a video that was posted on the VMware vApp developer blog.  In this demo a user deploys a mulit VM, multi-tiered version of SugarCRM in just a few clicks with no need to even start up a VM Console. 

New Products, VMware vApps, vCloud, VMware