VirtualPro

I’m currently working with my colleagues on an upgrade of our VI 3.5 infrastructure to vSphere Enterprise Plus.  We have recently been mulling over some of the design elements we will have to consider and one of the ones that came up was virtual Distributed Switches (vDS).  We like the look of it,  it saves us having to configure multiple hosts with standard vSwitches and it also has some nice benefits such as enhanced network vMotion support, inbound and outbound traffic shaping and Private VLANs.

One of the questions that struck me was,  what happens if your vCenter server fails? what happens to your networking configuration? Surely your vCenter server couldn’t be a single point of failure for your virtual networking, could it?

Well I did a bit of digging about, chatted to a few people on twitter and the answer is no it would not result in a loss of virtual networking.  In vSphere vDS the switch is split into two distinct elements,  the control plane and the data plane. Previously both elements were host based and configured as such through connection to the host, either directly using the VI client or through vCenter. In vSphere because the control plane and data plane have been separated,  the control plane is now managed using vCenter only and the data plane remains host based.  Hence when your vCenter server fails the data plane is still active as it’s host based where as the control plane is unavailable as it’s vCenter based.

One thing I was not aware of was where all this vDS information is stored . Mike Laverick over at RTFM informed me that the central config for a vDS is stored on shared VMFS within a folder called the .dvsData folder. I’ve since learnt that this location is chosen automatically by vCenter and you can use the net-dvs command to determine that location. It will generally be on shared storage that all ESX hosts participating in the vDS have access to.  As a back up to this .dvsData folder a local database copy is located in /etc/vmware/dvsData.db which I imagine only comes into play if your vCenter server goes down or if your ESX host loses connectivity to the shared VMFS with the .dvsData folder.  You can read more about this over at RTFM

Interesting links if your considering VMware Distributed Switches

VMware’s demo video of vDS in action, for those who want to learn more about vDS

Mike Laverick’s great reasoning on whether you should use vDS or not

Eric Sloof’s vDS caveats and best practices article

VMware’s vSphere Networking white paper explaining new vDS features

VMware’s vSphere Networking white paper on vDS network migrations

Jason Boche’s very interesting article on a specific vCenter + vDS issue

ESX, vCenter vCenter, VSphere

I’ve just finished a piece of work upgrading a few hosts to ESX 3.5 Update 4 using the ESX deployment appliance.  As part of the upgrade I install version 8.2.0 of the HP management agents for ESX. Upon completion of the install I realised that I could not see the Qlogic HBA’s within the Insight Manager home page.

Upon investigation it appears that HP have removed the libraries from their software from version 8.0.0 onwards.  Now ideally I would have installed the libraries before installing HP Insight Manager as indicated by Arnim Van Lieshout in his great HP agent upgrade script post.  In my case I had already done the install, so in order to see the HBA’s and the connected HP Storage, I had to carry out the following steps.

I downloaded the relevant Qlogic API library for ESX 3.5 U4 from the following site  
http://support.qlogic.com/support/oem_product_detail_vmware.asp

I created a temporary folder in the /tmp partition called qlogiclib

I unzipped the contents and copied it over to the new folder using WinSCP.

I logged on to the service console and ran the following commands to stop the HP agents.

service hpsmhd stop and service hpasm stop

I then navigated to the /tmp/qlogiclib folder and ran the command ./Install.sh

I then ran the following commands in the order shown below to restart the HP agents.

service hpsmhd start and service hpasm start

To check if it worked I navigated to https://hostname:2381 and checked for the external storage connections entry under the storage section on the home page.

Finally I deleted the /tmp/qlogiclib folder as it was no longer required.

Now the same steps carried out with vSphere ESX 4 does not have the desired effect.  Even though you can use the HBA test script located in the Qlogic API library they will not show up in HP Insight manager (Much to my annoyance).  When I eventually find a solution I will of course provide an update.

ESX, VMware, vSphere esx 3.5, QLogic

An interesting problem occurred the other day with one of our older production ESX 3.0.2 hosts. For the first time with any ESX host we have the service console memory ran out,  this resulted in all VM’s becoming unresponsive and loss of service to our users.

Now these hosts were built a couple of years ago by a consultant and all had their service console memory set to the default value of 272MB. I’m in the process of upgrading all hosts to ESX 3.5 U4 and changing the memory levels to the maximum 800MB,  this particular host was due to be upgraded in the next 2 weeks.  Unfortunate timing!!

VMware support were as helpful as ever and informed my colleague to up the service console memory to 800MB.  My only concern was the fact that your swap space is meant to be twice your service console memory.  If the memory was only set to 272MB you can be sure that the swap partition wasn’t going to be set to 1600MB.

My colleague was having trouble finding out what size the swap partition was so I gave him a hand. First of all he was doing a df –k at the service console,  which shows him the named linux partitions but not the swap partition we were looking for.  To get information on all disks and partitions attached to the host we need to run fdisk – l

This command showed us the swap partition created was made up of 1044225 blocks, though we weren’t sure exactly what this equated to in MB.

I took a look at one of our newly built ESX 3.5 U4 hosts and compared it’s fdisk –l results to the scripts used to build it.  I quickly found that by dividing by 1024 you could get the size of the partitions.  So in this case the swap partition on the ESX 3.0.2 host was roughly 1GB which was less than the recommended 2 x console memory sizing.

On this occasion VMware support advised us that it should be OK as it was.  That coupled with the fact we are going to rebuild the server in the coming weeks was enough for us to call the case closed.

However what if we did want to change it? I’d always been taught that changing the swap partition after the host had been built usually meant a full rebuild.  However as I’ve been working my way through Scott Lowe’s Mastering Vmware vSphere 4 book I came across the steps to do it without a rebuild.  It’s always recommended to rebuild a host as opposed to take this action, however occasionally needs must.

first create a new swap file on an existing service console partition, the command below will create a 1.6GB within the path entered /path/to/

dd if=/dev/zero of=/path/to/swap.file bs=1024 count =1640144

Use the following command to turn this into a usable swap file

mkswap /path/to/swap.file

Now enable the swap file with the following command

swapon /path/to/swap.file

If you do try this, it is entirely at your own risk. I haven’t as I am planning to rebuild in the near future.  If I wasn’t I would probably have given this a shot just to put my mind at ease.

ESX, VMware, vSphere esx, Service Console, Swap partition, VSphere

For those of you that will have heard of Alan Renouf you will undoubtedly know of his talents in the dark art of VMware CLI / Powershell.  For those of you who don’t know him I suggest you check out his web site  to sample some of the many great articles and scripts he’s already produced.

His latest powershell creation has recieved a lot of attention in the last couple of days and with good reason. The Daily Report is a configurable script where you can set thresholds and variables such as snapshot age, datastore space free thresholds or number of days to look at for vCenter warnings and errors.  The script when run goes off and examines your Virtual  Infrastructure based on these variables and then proceeds to email you a nice html report on the following items.

·         VMs created in the  x number of days and who created them.

·         VMs deleted in the  x number of days and who deleted them.

·         Datastores which have less than x% of free space remaining.

·         VMs that have CD-Rom or Floppy drives connected.

·         VMs with no VMware Tools installed.

·         Snapshots that are older than x number of days.

·         Current state of vCenter Services.

·         vCenter events that have been logged in x number of days.

·         Windows events  on the vCenter server that relate to VMware.

·         Hosts in maintenance mode or a disconnected state.

Get yourself over  to Alan’s site and download a copy of the script and give it a try,  I did today and the results were enough for me to go ahead and implement this as a scheduled task.  If you’d like to see more features in Alan’s Daily Report script then give him some feedback,  there are a few good suggestions on the blog post already and I’m sure the next version isn’t far away.  Great work Alan, keep it up!

ESX, VI Toolkit / Powershell, VMware, vCenter esx, vCenter, VI Toolkit / Powershell

I was purusing twitter the other day (as you do) when I came across a link shared by @Stevie_Chambers of VMware.  The link was for a new free tool from EMC called Compliance Checker for VMware ESX, which as you might guess does exactly what  it’s title suggests.  The tool though EMC branded is actually by Configuresoft who EMC bought outright earlier this year following an existing OEM relationship.

Now some of you may be familiar with the Tripwire Config Check tool which allows you to scan your ESX servers security configuration, which in itself is a very handy tool.  This new tool from EMC appears to be a little more comprehensive in that it checks your ESX servers against both the VMware VI3 Security Hardening Guidelines  and the Centre for Internet Security VMware ESX Server 3.x benchmark.  Test results against both are backed with an extensive knowledge base of what the check is and the actions required to rectify the issue.

Included below are some screenshots of the Main interface, the reports returned and the knowledgebase articles you recieve when you click on any item in the report.

I must admit that these compliance checkers are handy because my Linux experience only began with ESX and sometimes the areas of security being checked are ones I wouldn’t have a clue about.  This is where the knowledgebase is great because it explains a little about what the check actually is as well as the actions required to rectify the issue.  Very handy because it would appear that a standard build of ESX 3.5 U4 is only 73% compliant as far as this piece of software is concerned.

To get your free copy of Compliance Checker for VMware ESX, click the link,  you’ll need to register but it’s a small price to pay for this handy tool.

ESX, New Products, VMware EMC, esx

As soon as vSphere was put on General release I downloaded a copy of vSphere 4 ESXi for running on my home lab setup.  I’ve only recently started my home lab and the first machine I purchased was the HP Proliant ML115 G5 server. This was following a recommendation by Kiwi Si over at www.techhead.co.uk  who has extensively blogged about the suitability of the HP ML110 G5 and ML115 G5 for ESX labs.

If your interested in starting a home lab I can thoroughly recommend these HP server.  Simon even has a deal going with ServersPlus.com (UK) for free delivery on either the HP ML110 or ML115 server.  Get over to his hot deals page for further details.

Simon also has some great articles on getting ESXi 3.5 running from a USB pen drive.  This was a perfect place for me to start as I wanted to take advantage of the ML115 G5’s internal USB port and boot my server from a USB pen drive. So where do you start?

For those of you using an Apple Mac and wanting to conduct this excercise, check out the following article over on Tom Rowan’s blog

What do I need?

A USB Pen Drive that is over 1GB in capacity - nice and cheap at amazon
Download the vSphere ESX4i ISO image from here 
Download Shareware version of Winrar from here
Download Free Trial version of Winimage from here

How do I do it?

Once you have downloaded the ISO image open it up with WinRAR,  make sure you use WinRAR as I had problems with WinZip and UltraISO

Double click on the image.tgz file to open the contents in WinRAR and drill down to the \usr\lib\vmware\installer directory.

Within this folder you will find a file called VMware-VMvisor-big-164009-x86_64.dd.bz2. This is another zip file so double click on it and the contents will be displayed in a seperate WinRAR window.

Once inside extract the file VMware-VMvisor-big-164009-x86_64.dd using WinRAR and copy it somewhere locally on your PC.

Now install and open up the WinImage trial that you downloaded at the start of this process.

Insert your USB key and then select Disk and restore virtual hard disk image on physical drive as per the screenshot below.

 Select the physical USB drive from the list and click OK,  when prompted for the virtual disk file navigate to the dd file you extracted to your local PC.  This will now image your USB Key with the vSphere ESX4i hypervisor.

Once complete stick the USB key in a server / whitebox that supports 64 bit computing and away you go.  The screenshot below shows my own HP Proliant ML115 G5 running vSphere ESX 4i and all this from a simple 2GB USB pen drive.

ESX, VMware, vSphere esx4i, esxi, VSphere

I’m currently working on replacing all our ESX 3.0.2 and 3.0.1 hosts with 3.5 U4 hosts,  unfortunately the leap to vSphere is going to have to wait until next year.

One of my main reasons for doing this was to allow for the introduction of VMware Update Manager for ESX Host patching.  ESX Hosting patching can be a bit of a pain when using the service console and Esxupdate and because of this patching starts to become a nightmare and gets forgotten about.

So I had my first ESX 3.5 host added to virtual center and was poking around the VMware Update Manager tab and noticed that there were absolutely no host patches attached to the default host baselines.  I did a bit of digging around online and plenty of people had problems out there with the actual installation of VUM or their vci-integrity.xml file was missing some of it’s content.

Luckily my issue was resolved far more easily.  My virtual center server didn’t appear to be able to access the websites to get the patches due to the enhanced security pack.  All I had to do was add the following websites into the trusted sites on Internet Explorer on the Virtual Center server.

https://www.vmware.com
https://xml.shavlik.com

When the next scheduled patch download occured I recieved 150 patches for ESX 3.5.0 and was then able to create a custom baseline for all patches after 3.5.0 U4 and apply them.  Worked an absolute treat.

ESX, ESXi, VMware VMware Update Manager, VUM

Today is the day, VMware is now available for download and some really good news is that VMware are offering a 60 day evaluation trial of vSphere Enterprise Plus and vCenter Server Standard.

Three links for you, the first is the VMware vSPhere Download page and the second is the free 60 day evaluation download link.  For those who want to use the free version of ESX 4i and have a computer / server capable of supporting 64 bit you can get your free copy at this link.

Documentation for the various vSphere components can be found here http://www.vmware.com/support/pubs/vs_pubs.html

Credit to Michael Hany over at www.hypervizor.com for the links to these awesome video’s below, which should help you get to grips with some of the new vSphere 4 features.

ESX Installation and Configuration
ESXi Installation and Configuration
VMware vCenter Server
VMware vSphere Client
Networking configuration
Storage configuration (iSCSI)
Create and manage virtual machines
VMware Host Profiles
VMware Storage VMotion
VMware vCenter Server Linked Mode
VMware vNetwork Distributed Switch (vDS)

ESX, New Products, VMware, vSphere esx, esxi, VSphere

So I had the chance to play about with the latest version of Bluebear Kodiak the other day,  I just hadn’t had the chance to put some of my findings and thoughts down on the blog.

This release has been a long time in coming,  so much so that I’d almost forgotten what was in 0.0.2.  I often found myself looking at 0.0.3 and asking myself, “was that in it already or is that new?”

Most of the changes that have occured appear to be under the hood, one of these changes is the introduction of a Lua scripting engine.  I had never heard of Lua as a scripting language before this release,  however following a bit of reading it appears to be a very lightweight and fast scripting language that is highly customisable to requirements.  To be honest I have no idea what this will mean for Kodiak from an operational perspective,  it doesn’t appear to be any faster than the previous version but I was only testing this against a single ESX server.  The release notes hint that each server gets it’s own independent Lua script which may indicate that the benefits only appear when using Kodiak against multiple hosts.

I notice that support for Citrix XenServer and Microsoft Hyper-V are still missing in this release,  with no real indication as yet of a timeline for support of these mainstream hypervisors.  I’m very interested to see if and how it would handle managing multiple hypervisors within the same application window,  I’m presuming that it’s what they’re working towards and it would be a masterstroke if they can pull it off.  Microsoft have obviously tried something similar by managing VMware hosts within System Center Virtual Machine Manager (SCVMM) with mixed results,  lets hope the guys at BlueBear have more joy in their attempts of building a management tool to bridge all mainstream hypervisors.

So,  lets get down to the nice bits.  First off the map, a great interface and easy to use, move about and navigate.  This release sees the addition of a new map connection node and link display control which allows you to customise what’s shown on the map to good effect.  It can get very busy on screen sometimes and this feature allows you to control exactly what you do and don’t display in a very granular way.  I believe the scripts window is a new edition,  when a machine is selected you are presented with an array of scripts to use including defragment vm disks and migrate reboot to BIOS. From the release notes it appears that users will be able to create their own scripts however a user interface / script builder still needs to be added

I must admit I did have the occasional bit of trouble when navigating round the interface.  I seemed to be able to open the console window without issue by clicking on the appropriate button when a host was selected.  However I was having issues opening the configuration screen in that sometimes it just did not respond at all.  I also saw issues with VM status refresh as well,  I powered down a linux box by issuing a shutdown within the OS. However Kodiak still reported it as switched on within the map section,  a minor annoyance of course but a core feature has to be the accurate reporting of virtual machine status.

It looks like the guys at Bluebear have been working hard on changing some of the core fundamentals in the background,  as such there aren’t maybe as many differences on top as you might expect.  What I’m hoping is that the work done between 0.0.2 and 0.0.3 underpins their master plan and that we’ll start to see development pick up. We hope to see the additional hypervisor support, the continued fine tuning of the interface and of course delivery of features to meet the requirements people will undoubtedly have in their virtualisation management tool of choice.

I don’t have any invites left but Gerald Bunch over at Professionalvmware.com has quite a few left

ESX, ESXi, New Products bluebear, ESX management

The guys over at Bluebear have released version 0.0.3 of their Kodiak management tool, aptly named the boisterous bruin.  You can download your copy here if your a member of the private beta program and you can view the full release notes here. 

I’ve installed it and had a quick look, appears to be a lot of new additions so will wade through them all and post back with a more detailed update later tonight.

This version of Kodiak represents a significant change in our server-communication architecture.

We’ve added a schema-compliant request/argument builder engine that populates requests based on known information. This makes building requests much easier.

A lua scripting engine. Each server connection gets its own independent Lua script.

Independent object pools.

Improved mapping response and control.

Map connection node & link display control.

We’ve created a “plugin api” where a user may build a plugin to communicate with any kind of data service. We’ll be publishing information on this on our wiki.

We’ve added a number of “boilerplate” features that will become more apparent in the next few releases as we push Kodiak towards our goal of creating an “IT IDE”.

Access for users to create their own scripts. Kodiak’s UI/inspector controls are driven by a “global schema” that allows a user to create their own customized scripts for management. Please visit the bluebear wiki for further information regarding customizing the script engine. As yet, we don’t have a full UI for managing user scripts, but we will shortly.

The remote event log has been removed for this release.

For Windows users who experience the dreaded “SSL certificate bug”, we’ve added a preferences option to bypass the windows SSL stack. This option is accessible from the Preferences menu.

Password vault & credentials storage. Kodiak now has the ability to save username/password pairs in an encrypted password vault, locked with a master password.

Server object creation is disabled while we migrate the control components into the VMWare plugin, we will re-enable these in a couple of days.

ESX, New Products bluebear, ESX management, kodiak