VirtualPro

So I had the chance to play about with the latest version of Bluebear Kodiak the other day,  I just hadn’t had the chance to put some of my findings and thoughts down on the blog.

This release has been a long time in coming,  so much so that I’d almost forgotten what was in 0.0.2.  I often found myself looking at 0.0.3 and asking myself, “was that in it already or is that new?”

Most of the changes that have occured appear to be under the hood, one of these changes is the introduction of a Lua scripting engine.  I had never heard of Lua as a scripting language before this release,  however following a bit of reading it appears to be a very lightweight and fast scripting language that is highly customisable to requirements.  To be honest I have no idea what this will mean for Kodiak from an operational perspective,  it doesn’t appear to be any faster than the previous version but I was only testing this against a single ESX server.  The release notes hint that each server gets it’s own independent Lua script which may indicate that the benefits only appear when using Kodiak against multiple hosts.

I notice that support for Citrix XenServer and Microsoft Hyper-V are still missing in this release,  with no real indication as yet of a timeline for support of these mainstream hypervisors.  I’m very interested to see if and how it would handle managing multiple hypervisors within the same application window,  I’m presuming that it’s what they’re working towards and it would be a masterstroke if they can pull it off.  Microsoft have obviously tried something similar by managing VMware hosts within System Center Virtual Machine Manager (SCVMM) with mixed results,  lets hope the guys at BlueBear have more joy in their attempts of building a management tool to bridge all mainstream hypervisors.

So,  lets get down to the nice bits.  First off the map, a great interface and easy to use, move about and navigate.  This release sees the addition of a new map connection node and link display control which allows you to customise what’s shown on the map to good effect.  It can get very busy on screen sometimes and this feature allows you to control exactly what you do and don’t display in a very granular way.  I believe the scripts window is a new edition,  when a machine is selected you are presented with an array of scripts to use including defragment vm disks and migrate reboot to BIOS. From the release notes it appears that users will be able to create their own scripts however a user interface / script builder still needs to be added

I must admit I did have the occasional bit of trouble when navigating round the interface.  I seemed to be able to open the console window without issue by clicking on the appropriate button when a host was selected.  However I was having issues opening the configuration screen in that sometimes it just did not respond at all.  I also saw issues with VM status refresh as well,  I powered down a linux box by issuing a shutdown within the OS. However Kodiak still reported it as switched on within the map section,  a minor annoyance of course but a core feature has to be the accurate reporting of virtual machine status.

It looks like the guys at Bluebear have been working hard on changing some of the core fundamentals in the background,  as such there aren’t maybe as many differences on top as you might expect.  What I’m hoping is that the work done between 0.0.2 and 0.0.3 underpins their master plan and that we’ll start to see development pick up. We hope to see the additional hypervisor support, the continued fine tuning of the interface and of course delivery of features to meet the requirements people will undoubtedly have in their virtualisation management tool of choice.

I don’t have any invites left but Gerald Bunch over at Professionalvmware.com has quite a few left

ESX, ESXi, New Products bluebear, ESX management

The guys over at Bluebear have released version 0.0.3 of their Kodiak management tool, aptly named the boisterous bruin.  You can download your copy here if your a member of the private beta program and you can view the full release notes here. 

I’ve installed it and had a quick look, appears to be a lot of new additions so will wade through them all and post back with a more detailed update later tonight.

This version of Kodiak represents a significant change in our server-communication architecture.

We’ve added a schema-compliant request/argument builder engine that populates requests based on known information. This makes building requests much easier.

A lua scripting engine. Each server connection gets its own independent Lua script.

Independent object pools.

Improved mapping response and control.

Map connection node & link display control.

We’ve created a “plugin api” where a user may build a plugin to communicate with any kind of data service. We’ll be publishing information on this on our wiki.

We’ve added a number of “boilerplate” features that will become more apparent in the next few releases as we push Kodiak towards our goal of creating an “IT IDE”.

Access for users to create their own scripts. Kodiak’s UI/inspector controls are driven by a “global schema” that allows a user to create their own customized scripts for management. Please visit the bluebear wiki for further information regarding customizing the script engine. As yet, we don’t have a full UI for managing user scripts, but we will shortly.

The remote event log has been removed for this release.

For Windows users who experience the dreaded “SSL certificate bug”, we’ve added a preferences option to bypass the windows SSL stack. This option is accessible from the Preferences menu.

Password vault & credentials storage. Kodiak now has the ability to save username/password pairs in an encrypted password vault, locked with a master password.

Server object creation is disabled while we migrate the control components into the VMWare plugin, we will re-enable these in a couple of days.

ESX, New Products bluebear, ESX management, kodiak

There have been a couple of postings recently by people who have spent some time custom building PC’s capable of running VMWare ESX. A great example is Eric Sloof who has built a monster white box capable of running the new VI4 Beta software.

That’s got me interested in building one,  perhaps to road test new features if I can ever get invited to participate in a VMWare beta program,  or perhaps just to run ESX 3.5i so I can play about with new Microsoft Windows 7 or Server 2008 R2 Beta.

One of the main issues with building a whitebox is ensuring compatibility with the VMWare HCL,  so where on earth do you start looking so you can ensure your whitebox works.  Well I’ve put togethere some links together to get you started,  I hope they’re of some help.

VMWare Hardware Compatibility List

The new fully searchable HCL just has to be on the list,  double check your planned components,  you may be suprised to find they are actually supported on the HCL.

VM-Help.com - The unnofficial VMWare ESX whitebox HCL

This site is stacked full of useful information on tried and tested hardware combinations

The Ultimate Whitebox web site

Though this is a relatively new site it has some really well laid out information on hardware combos

Tim Jacobs post on his Whitebox

This post is just under a year old but still contains some good information.

Now HP were doing some crazy deals on their HP ML tower server range last year.  I was having a look at servers direct in the UK today and the prices are pretty good,  I’m tempted to buy one of these  rather than build a solution from scratch.

So how have people got on with ESX on the HP ML Series server? Well here are a few more links that may help.

instaling ESX 3.5 on the HP ML110 G5

Kiwi Si who runs the TechHead has a great lab article about installing ESX 3.5 on the ML110 G5

ESX 3.5 operating on a ML115 G5

Virtual VCP has put this article together detailing the changes required to get ESX 3.5 working on the new ML115 G5 servers.

Thing to remember with all whiteboxes are they are not supported by VMWare and you will be relying on the VMWare community to help you out with any problems.  My advice is,  if someone has listed the exact components that worked for them,  copy it as it will most likely work for you.  I’ll let you know how I get on with my whitebox,  just need to wait for that much needed January pay cheque ;o)

ESX, VMware ESX 3.5i, VMware

I follow a lot of people on twitter who write about virtualisation and have there own blog.  So up popped a message from Jason Boche over at www.boche.net about a new article he’d put up on the site about Vmotion performance.

Written by a gentleman called Simon Long it is a excellent article.  It just goes to shows what happens when something doesn’t work as quickly as you’d like,  some people have the determination and spend the time finding out why.  This post is a gem and one of those little tweaks that will come in very handy in the future

I’ll set the scene a little….

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, *********, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.

Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.

1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpdx.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;

<ResourceManager>
<maxCostPerHost>12</maxCostPerHost>
</ResourceManager>

5. Now you need to decide what value to give “maxCostPerHost”.

A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).

I am unsure on the maximum value that you can use here, the largest I tested was 24.

6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.

Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.

I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.

Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.

The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;

Section 1 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30

Section 2 - This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).

Section 3 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36

Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.

Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24

Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54

Time Different = 30secs

It’s still an improvement all be it not so big.

Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.

I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.

ESX, VMware, vCenter esx, VI3

Some time ago long before I started my blog I used an application called Trip Wire Config Check on some of my employers ESX Hosts. We are primarily a Microsoft software house so our technical experience of Linux / Unix was minimal before the introduction of ESX into our server estate. The config check application checks the current host configuration against the VI3 Security hardening guidelines reporting back any vunerabilities.

A some of the configuration parameters being checked include:

- Virtual network labeling
- Port Group settings
- Network isolation for VMotion and iSCSI
- NIC Mode settings / Layer 2 Security settings
- VMWare ESX Service Console security settings
- SAN resource masking and zoning

The results that the tripwire config check returned showed that we had a number of areas that needed to be addressed with our host security configurations. The results provided a great checklist of everything that was right and wrong with our current build procedures.  Some of the items that came up we would never have even considered,  some of them we felt were minor and could be ignored,  others needed to be addressed immediately.  I highly recommend taking a look at this free tool and running it against one of your standard host builds to see if you need to change it, hopefuly you won’t need to revist all your hosts.

Download your free copy of Tripwire - Config Check and check out the following blog post for installation and usage instructions.  

Tripwire® ConfigCheck^TM is a free utility that rapidly assesses the security of VMware ESX 3.0 and 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

ESX, New Products, VMware esx, security

I subscribe to a large number of virtualisation feeds and the primary reason is that I never fail to learn something new every single day.

I came across a blog entry by the excellent Duncan Epping over at Yellow Bricks about VMFS Heap size configuration.  I was not previously aware of this setting because we have 3.0.x in our environment and our HP MSA storage only support 3.6Gb fully populated so this scenario would never come up.  I will however be keeping this in mind when designing our new ESX 3.5 environment.  We seem to be utilising bigger hosts all the time and when you add in a new EMC Clariion CX4, the potential for one host to be accessing more than 4TB of open VMDKs goes up.

was talking to a fellow consultant today. He ran into the following error messages at one of his customer sites:

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0×8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

During the conversation I knew I’d seen this problem before. But the problem that I witnessed was related to a high threshold value in Vizioncore vFoglight. I knew it was possible to change the setting:

1. Open vCenter, and click a specific host
2. Click on the “Configurations” tab
3. Click on Advanced Settings, VMFS3
4. Change the value of “VMFS3.MaxHeapSizeMB”

 

The default value is 16MB, this allows for a maximum of 4TB of open vmdk’s on a single host. The max setting is 128MB which allows for a maximum of 32TB of open vmdk’s on a single host. Keep this in mind when designing your environment.

Keep in mind that this is ESX 3.5 only, you can’t change the heap size in ESX 3.0.x.

ESX, VMware esx, esx 3.5, VMware

I read an interesting blog post on Mike De Petrillo’s blog entitled Microsoft Lies to their customers again.  The article details a discussion with a customer regarding licensing on virtual platforms such as ESX and Hyper-V. 

Customer: I would love to use VMware but the cost savings in Windows licenses from Microsoft with Hyper-V makes it a deal I can’t pass up.

Me: What do you mean by that? Licenses cost you the same for Windows no matter what virtualization solution you’re using.

Customer: That’s not what my Microsoft rep told me. They said I could get unlimited virtual machines with Datacenter Edition of Windows only if I used Hyper-V.

Me: Ah. I see. You know, they’re lying to you, right?

Customer: No. They never lie to me. Where’s the proof.

Now I’ve done a lot of work with company reps from all sorts of companies in the past.  The thing to remember is these guys are sales guys pure and simple,  they are interested in getting your business for their company, bottom line.  If you say “I need to consolidate my server pool and keep the cost down”,  the Microsoft Rep is obviously going to tell you that Microsoft Hyper-V is going to be the best and cheapest way to achieve this.  Fair enough that’s his opinion,  but as an IT Professional it is your job to see through the smoke and mirrors and substantiate any claims made.  I take everything I hear with a pinch of salt (even from VMware) and I will always try to validate any claim made, either through existing customer reference sites, product forums or blog sites. 

I’m still reeling from the fact that Mike is seeing this across a number of customers,  how hard can it be to do a little research.  I did a very quick google search on “licensing for virtualization“ and the top result was the press release relating to Microsoft’s change of licensing to incorporate virtualisation.  The second result was the Microsoft page that details licensing for specific products such as Windows Server 2008 and SQL Server 2008.

Within that page is a link to the white paper I used to clarify the correct approach to be taken by my current employer.  The first paragraph answers the question this guy in New York should have been asking, don’t you think?

 The purpose of this white paper is to give an overview of Microsoft® licensing models for the server operating system and server applications under virtual environments. It can help you understand how to use Microsoft server products with virtualization technologies, such as Microsoft Hyper-V™ technology, Microsoft® Virtual Server 2005 R2, or third-party virtualization solutions provided by VMWare and Parallels.

ESX, ESXi, Microsoft, VMware esx, hyper-v, licensing, VMware, windows 2008

I was looking into an issue following an upgrade to vCenter Server 2.5 last weekend.  So I set about searching through the file system for the log files on the server with very little luck to be honest.

I then found two excellent posts from Rick Blythe a.k.a the vmwarewolf,  the posts detail the locations of the logs and what each one means.  This is an excellent post and one that I’m going to keep handy for all those strange little issues where insight into the logs might give a clue to the problem.

Virtual Center Logs
http://www.vmwarewolf.com/which-virtual-center-log-file/

ESX Server Logs
http://www.vmwarewolf.com/which-esx-log-file/

ESX, ESXi, VMware, vCenter esx, vCenter, virtual center

Well BlueBear have kindly given me another 47 beta invites to give away to coincide with the release of the second beta version of the software , Interestingly named “Curious Cub”.

I haven’t had a chance to take a look at it myself but have had a good look at Duncan Epping’s article over at Yellow Bricks which you can read by clicking here

If you’d like a beta invite just drop me a comment on this post and I will do my best to process them all tomorrow.  Might be after the day job has finished though :o)

Beta Release Documentation

New functionality

We have added support for configuring existing virtual machines. All the expected functions are available. Configuration for a Virtual Machine is accessible via the configure button attached to a given VM object on the screen (Inspector Panels / Map)

Configuration of all standard resources is available, e.g. Memory Size, CPU allocation/shares, virtual disks, virtual NICs, and removable devices. Usage Notes

Virtual disks are of the “preallocated/thick” type, support for all disk types will be available in the next release.

‘Host-device-backed’ virtual devices (e.g. floppy/cdrom/raw) are not supported, but will be available in a future release.

‘Remote-backed’ virtual devices (e.g. floppy/cdrom/ISO images on the client) are not supported, but will be available in a future release.

VMs must be powered off for configuration. While it is possible to reconfigure a running VM, it may result in a host fault that will cause your changes to be discarded.

If you make changes to a VM that you don’t want to commit, simply close the configure window and the changes will be discarded, (or hit the discard button).

Support for VM Creation will be available in the next release.

Known Issues

Untrusted certificates on windows may cause an authorization dialog to continually appear. The certificate must be added to the local store, please see this page for a work-around.

License keycodes are case sensitive, please ensure you enter them exactly.

VirtualCenter is not yet fully supported. While it might work, functionality is not guaranteed to give predictable results. We are currently adapting Kodiak’s back-end data model to more effectively handle VirtualCenter’s data structures.

Support / Bug Reports

We’re working on enhancing our support wiki, bug reporting and additional content are almost ready.

ESX, General, New Products bluebear, ESX management

SolarWinds have announced a handy little desktop application for monitoring the status of a single ESX Host and it’s VM’s.  You can view a video overview of the product on YouTube by following the link below.

http://nl.youtube.com/watch?v=KuqWrfGvNoI&feature=related 

You can download this little piece of software for free from the following link (you do have to register though)

http://www.solarwinds.com/products/freetools/vm_monitor.aspx

ESX, General, VMware esx, management, monitoring