VirtualPro

It would appear that with the upcoming release of VMware vSphere (VI4) there will be an update to the VMware Certified Professional (VCP) certification.

Eric Sloof over at NTPRO.NL has posted an article on his site about this news.  He references a post on the vmware communities site which has the following comments from John Hall (Technical Certification Developer)

There is no time limit between when you take the class and when you take the exam. The only thing I will say is that at some point this year we will announce a new VCP certification that will apply to the new release of our datacenter product. The training you took will apply to the VCP on VI3 certification exam, but there may be some additional requirements in order to apply your training to the newer certification. Even after we announce the new certification, you can still achieve the VCP on VI3 certification using your existing training.

Jon C. Hall
Technical Certification Developer

Certifications, VI4, VMware VCP, VMware, VSphere

I follow a lot of people on twitter who write about virtualisation and have there own blog.  So up popped a message from Jason Boche over at www.boche.net about a new article he’d put up on the site about Vmotion performance.

Written by a gentleman called Simon Long it is a excellent article.  It just goes to shows what happens when something doesn’t work as quickly as you’d like,  some people have the determination and spend the time finding out why.  This post is a gem and one of those little tweaks that will come in very handy in the future

I’ll set the scene a little….

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, *********, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.

Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.

1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpdx.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;

<ResourceManager>
<maxCostPerHost>12</maxCostPerHost>
</ResourceManager>

5. Now you need to decide what value to give “maxCostPerHost”.

A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).

I am unsure on the maximum value that you can use here, the largest I tested was 24.

6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.

Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.

I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.

Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.

The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;

Section 1 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30

Section 2 - This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).

Section 3 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36

Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.

Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24

Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54

Time Different = 30secs

It’s still an improvement all be it not so big.

Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.

I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.

ESX, VMware, vCenter esx, VI3

Some time ago long before I started my blog I used an application called Trip Wire Config Check on some of my employers ESX Hosts. We are primarily a Microsoft software house so our technical experience of Linux / Unix was minimal before the introduction of ESX into our server estate. The config check application checks the current host configuration against the VI3 Security hardening guidelines reporting back any vunerabilities.

A some of the configuration parameters being checked include:

- Virtual network labeling
- Port Group settings
- Network isolation for VMotion and iSCSI
- NIC Mode settings / Layer 2 Security settings
- VMWare ESX Service Console security settings
- SAN resource masking and zoning

The results that the tripwire config check returned showed that we had a number of areas that needed to be addressed with our host security configurations. The results provided a great checklist of everything that was right and wrong with our current build procedures.  Some of the items that came up we would never have even considered,  some of them we felt were minor and could be ignored,  others needed to be addressed immediately.  I highly recommend taking a look at this free tool and running it against one of your standard host builds to see if you need to change it, hopefuly you won’t need to revist all your hosts.

Download your free copy of Tripwire - Config Check and check out the following blog post for installation and usage instructions.  

Tripwire® ConfigCheck^TM is a free utility that rapidly assesses the security of VMware ESX 3.0 and 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

ESX, New Products, VMware esx, security

I’ve recently been looking at posting something on SRM as there seems to be so much information out there. I will hopefully find the time to write a post that summarises everything I’ve seen so far but I thought this would be a good starter for ten.  Richard Garsthagen over at run virtual has posted a new official VMware video introduction to VMware Site Recover Manager (SRM),  You can view it below.

He also included a handy link to a document which tells you how to setup a full 2 site SRM demo suite with ESX on a single laptop.  Handy for setting up a demo lab if you want to get familiar with the product or if your’re a VMware partner you may want to set this up for a sales demonstration.

http://viops.vmware.com/home/docs/DOC-1235

New Products, SRM, VMware esx, SRM, VMware

I subscribe to a large number of virtualisation feeds and the primary reason is that I never fail to learn something new every single day.

I came across a blog entry by the excellent Duncan Epping over at Yellow Bricks about VMFS Heap size configuration.  I was not previously aware of this setting because we have 3.0.x in our environment and our HP MSA storage only support 3.6Gb fully populated so this scenario would never come up.  I will however be keeping this in mind when designing our new ESX 3.5 environment.  We seem to be utilising bigger hosts all the time and when you add in a new EMC Clariion CX4, the potential for one host to be accessing more than 4TB of open VMDKs goes up.

was talking to a fellow consultant today. He ran into the following error messages at one of his customer sites:

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0×8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

During the conversation I knew I’d seen this problem before. But the problem that I witnessed was related to a high threshold value in Vizioncore vFoglight. I knew it was possible to change the setting:

1. Open vCenter, and click a specific host
2. Click on the “Configurations” tab
3. Click on Advanced Settings, VMFS3
4. Change the value of “VMFS3.MaxHeapSizeMB”

 

The default value is 16MB, this allows for a maximum of 4TB of open vmdk’s on a single host. The max setting is 128MB which allows for a maximum of 32TB of open vmdk’s on a single host. Keep this in mind when designing your environment.

Keep in mind that this is ESX 3.5 only, you can’t change the heap size in ESX 3.0.x.

ESX, VMware esx, esx 3.5, VMware

I read an interesting blog post on Mike De Petrillo’s blog entitled Microsoft Lies to their customers again.  The article details a discussion with a customer regarding licensing on virtual platforms such as ESX and Hyper-V. 

Customer: I would love to use VMware but the cost savings in Windows licenses from Microsoft with Hyper-V makes it a deal I can’t pass up.

Me: What do you mean by that? Licenses cost you the same for Windows no matter what virtualization solution you’re using.

Customer: That’s not what my Microsoft rep told me. They said I could get unlimited virtual machines with Datacenter Edition of Windows only if I used Hyper-V.

Me: Ah. I see. You know, they’re lying to you, right?

Customer: No. They never lie to me. Where’s the proof.

Now I’ve done a lot of work with company reps from all sorts of companies in the past.  The thing to remember is these guys are sales guys pure and simple,  they are interested in getting your business for their company, bottom line.  If you say “I need to consolidate my server pool and keep the cost down”,  the Microsoft Rep is obviously going to tell you that Microsoft Hyper-V is going to be the best and cheapest way to achieve this.  Fair enough that’s his opinion,  but as an IT Professional it is your job to see through the smoke and mirrors and substantiate any claims made.  I take everything I hear with a pinch of salt (even from VMware) and I will always try to validate any claim made, either through existing customer reference sites, product forums or blog sites. 

I’m still reeling from the fact that Mike is seeing this across a number of customers,  how hard can it be to do a little research.  I did a very quick google search on “licensing for virtualization“ and the top result was the press release relating to Microsoft’s change of licensing to incorporate virtualisation.  The second result was the Microsoft page that details licensing for specific products such as Windows Server 2008 and SQL Server 2008.

Within that page is a link to the white paper I used to clarify the correct approach to be taken by my current employer.  The first paragraph answers the question this guy in New York should have been asking, don’t you think?

 The purpose of this white paper is to give an overview of Microsoft® licensing models for the server operating system and server applications under virtual environments. It can help you understand how to use Microsoft server products with virtualization technologies, such as Microsoft Hyper-V™ technology, Microsoft® Virtual Server 2005 R2, or third-party virtualization solutions provided by VMWare and Parallels.

ESX, ESX 3.5i, Hyper-V, Microsoft, VMware, Windows 2008 esx, hyper-v, licensing, VMware, windows 2008

VMware have announced that VI4 will be renamed VMware vSphere going forward.  Jason Boche of boche.net heard the news at a VMware User Group meeting in Minneapolis and following a quick check with the powers that be at VMware he was given the go ahead to release the news.

Eric Sloof over at NTPRO.NL has however found out that a product already exists with this name. I myself found this company who appear to have the existing trademark on vSphere.  Nothing a fat cheque from VMware can’t resolve I’m sure.

Changing the name of your products, seems to be a popular thing to do at the moment.  Citrix change presentation server to XenApp, Microsoft changed SoftGrid to App-V.  VMware not one to miss out on the action have now changed Virtual Center to vCenter Server and VI4 to vSphere.  We await the official announcement from VMware on the name change and whether it’s going to end there.

New Products, VI4, VMware VI4, VMware, VSphere

I was looking into an issue following an upgrade to vCenter Server 2.5 last weekend.  So I set about searching through the file system for the log files on the server with very little luck to be honest.

I then found two excellent posts from Rick Blythe a.k.a the vmwarewolf,  the posts detail the locations of the logs and what each one means.  This is an excellent post and one that I’m going to keep handy for all those strange little issues where insight into the logs might give a clue to the problem.

Virtual Center Logs
http://www.vmwarewolf.com/which-virtual-center-log-file/

ESX Server Logs
http://www.vmwarewolf.com/which-esx-log-file/

ESX, ESX 3.5i, VMware, vCenter esx, vCenter, virtual center

Well this weekend I had the job of updating our Virtual Center deployment from 2.0.2 to 2.5 update 3. The primary reason for this was to prepare for the introduction of ESX 3.5 hosts into the Virtual infrastructure.

Due to the importance of our Virtual infrastructure I decided to do as much reading and preperation as possible to ensure it all went smoothly. It’s easier to convince our change management team to let us make changes if we get them right first time,  so this one was important to facilitate an easier path for future change.

So what did I read to ensure I’d covered everything,  here are a few links to get you started

Mike Laverick’s Upgrade Experience PDF
http://www.rtfm-ed.co.uk/?p=482

vCenter Server 2.5 Update 3 - Release Notes
http://www.vmware.com/support/vi3/doc/vi3_vc25u3_rel_notes.html

ESX 3.5 and vCenter Server 2.5 Upgrade Guide
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_upgrade_guide.pdf

The above PDF is a brilliant guide to the process you should follow, including rollback.  You should read this thoroughly so you understand all the pre-requisites and can avoid those silly problems that could cause your upgrade to fail.

Of course vCenter Server 2.5 introduces Update Manager. Although we can’t use it as we have ESX 3.0.1 and 3.0.2 hosts ( supports 3.0.3, 3.5 and 3.5i only ) I decided to install it anyway so it’s there for the future.

Here are some of the links I used to plan out my update manager deployment.

Update Manager Administration Guide
http://www.vmware.com/pdf/vi3_vum_10_admin_guide.pdf

Update Manager Performance and Best Practice Guide
http://www.vmware.com/pdf/vum_1.0_performance.pdf

Update Manager Size Estimator
http://www.vmware.com/support/vi3/doc/vi3_vum_10_sizing_estimator.xls

One of the main mistakes that people tend to make is to not give the SQL accounts the correct permissions on the new update manager database and the MSDB database.  Make sure you cover this one or you upgrade will fail.

It all went quite smoothly, I initially had a couple of issues which appeared to be related to me attempting to do a custom install. I wanted to ensure I could go through all settings and customise as required, the install however failed with various MSI error messages. I started the install again and didn’t choose the custom setup this time. This however created an issue whereby the Update Manager database appeared to install as SQL Server 2005 Express.  I wanted to put it on the same SQL 2000 server as our Virtual Center database but I never got the option as far as I can remember.  I have today uninstalled Update Manager and re-installed it using the “use an existing database” option and the SQL 2000 database.  It worked fine the second time around.

No immediate problems following the upgrade,  I had read some horror stories about issues with the Virtual Centre Agent on the host not updating.  Luckily for me it wasn’t an issue. Good luck with your upgrade

VMware, vCenter vCenter, virtual center, VMware

VMWare have released a new search utility for searching their hardware compatibility list.  This replaces the old PDF versions which were a bit of a nightmare to wade through.  I’ve just tried it using some components that I had recently looked up in the old PDFs and I have to say I like it,  much easier to use and much clearer.

You can use the new searchable hardware compatability guide by clicking the link. 

General, VMware VMware