VirtualPro

I was doing some work out of hours the other night on my employers Virtual Infrastructure when bang on time the little red triangles started popping up against certain ESX hosts in vCenter.  Why you ask? well it’s AV scanning time on our VM’s of course, or the Sophos summit as we affectionately call it due to its uncanny resemblance to a mountain range when you look at the CPU performance stats in vCenter.

It got me thinking, has any one vendor actually got a product out there utilising the VMSafe API that could help me rid our virtual infrastructure of this problem?

My first stop was of course the main VMSafe page where I did find a large list of official partners who are working on developing products to utilise the VMSafe API. The pleasing thing to see was that there are plenty of mainstream security vendors taking part.  However I’ve still to see any of them releasing a product to market that actually utilises VMSafe.

Earlier this year in Glasgow I heard Mcafee talk about VMSafe as part of the VMware vSphere launch road show.  They talked about building a vApp that could sit in your Virtual Infrastructure and take care of AV scanning with the aim of reducing the CPU overhead that AV scanning introduces. I did a little trawl of the web and couldn’t find anything official, I did however find the following forum post (quoted below) which is definitely the unofficial line.

Virus Scan for Offline images is available, which uses VMSafe APIs to scan offline disks accessed via ESX

Nothing is currently road mapped for on-access scanning - no AV vendor has this technology available (or even road mapped as far as I’m aware) yet.

I did a bit more digging on this “scan offline disks” comment and found a recent article by VMware’s Richard Garsthagen.  This article reveals that a piece of software called the VMware Virtual Disk Development Kit (VDDK) can be used to conduct an offline scan of disks attached to powered on or off virtual machines (quoted below). 

VMware VDDK (also being seen as part of the VMsafe initiative, but has been available for longer). The VDDK is an disk API, that allows other programs to access a virtual machine’s hard disk like the VMware Consolidated Backup solution does. It does not matter is the VM is powered on of off, but a disk can just be ‘extra’ mounted to another virtual machine that for instance runs a virus scanner. The clear downside of VDDK is that nothing is real time.

Surely this would rid me of my daily scheduled Sophos summit, wouldn’t it? Think of a hypothetical scenario where you have a VDI setup with 1000 windows XP VM’s,  imagine the strain put on your ESX clusters by 1000 machines kicking off a scheduled daily AV Scan. Would an appliance that could offline scan disks reduce the strain? Well thinking about it, possibly not.  It would still have to conduct a scan of 1000+ virtual disks, only this time it wouldn’t have nearly as many CPU cycles available to churn through the work. All it would have is the resources assigned to the vApp which is likely to be completely inadequate for such a large task. With this in mind it’s likely that it would probably take a large amount of time to complete.  It could even take longer than a day which wouldn’t be much use for a daily AV scan. I’m sure some companies would rather suffer the ESX CPU resource pain point as opposed to sacrificing security through ineffective or untimely AV Scanning.

Richard’s article along with the solutions tab on the VMSafe webpage did however reveal that a couple of products that use VMSafe have made it to market.  One is called vTrust from Reflex Systems which appears to be a multi faceted application, which according to their site provides dynamic policy enforcement and management, virtual segmentation, virtual quarantine and virtual networking policies.  The other application is a hypervisor based firewall appliance from Altor that supports virtual segmentation and claims to provide better throughput by using the Fast Path element of the VMSafe API.

So it would appear on the surface that progress has been slow.  To only find two VMware certified appliances in the market place was, I have to admit, quite a surprise!  It looks like it’s going to be a while before we see VMsafe being fully utilised by vendors, even then we will  have those wary individuals who will never quite be convinced.

Neil Macdonald of Gartner makes a good point about the potential for VMSafe appliances to introduce possible security vulnerabilities at a lower level in the infrastructure.

If I’m responsible for VM security, I’ll consider it after the APIs ship, after the vendors finally ship their VMSafe-enabled solutions, after I’ve got a level of comfort that these VMSafe-enabled security solutions don’t in of themselves introduce new security vulnerabilities

Edward L Haletky who is very much focused on virtualisation security also makes a good point about low level vulnerabilities and the interaction of multiple VMSafe appliances. 

I fully expect VMware to not only ensure the VMSafe fastpath drivers do nothing harmful to the virtual environment, but also address interaction issues between multiple VMSafe fastpath drivers. In addition, I would like such reports made available to satisfy auditing requirements.

So was VMSafe simply something to bolster the vSphere marketing launch,  an announcement made before it should have been?  Usually VMware are quite good at keeping these kind of things under wraps and releasing them when they are a little more mature and ready for use in real world scenarios.  Now I don’t know what work was done with partners in advance but I would have liked to have seen a couple of the major security vendors releasing appliances at the same time as VMSafe was announced.  For me that certainly would have installed a little more confidence in VMSafe than writing this article has.

If anyone out there is writing appliances utilising the VMSafe API and wants to comment, please do.  I would love to hear some news from the front line as to what is being developed, where it will be applied and when we can expect to see it.

General, Gestalt-IT, vSphere VMsafe, VSphere

Over the last month or so I’ve had two invites to participate in vendor events abroad.  The first was an invite to the Gestalt IT tech day in San Francisco, the second was an invite to the EMC EMEA Customer Council event in Prague.  Now as much as I would love to go to everything I get invited to, I have a day job which pays the bills so in this instance I had to chose the one most relevant to my employer and that was the EMC EMEA Customer Council.

Having never been invited to an EMC Customer Council event before I wasn’t entirely sure what to expect. The basic structure of the event involved EMC sharing product roadmap and strategy, deep diving a few key technologies / strategies and then listening to customer feedback.  The sessions I attended were very interactive round table discussions, with a lot of enterprise customers who were not backward in coming forward with their feelings and opinions. As the sessions went on I started to see why EMC run these events. It would be hard to gain this kind of candid and honest feedback through any other medium, this kind of information is invaluable to a vendor. From my perspective as a customer I got a lot of good insight into roadmap, allowing me to more accurately propose a long term EMC storage strategy for my employer.  I also got to meet and chat to a lot of interesting people and best of all, I got to hear about the experiences of other customers. It was re-assuring to hear that whether you are an SMB IT operation or an enterprise level one, you tend to have very similar issues. The only difference sometimes being the scale of the infrastructure involved.

Now unfortunately unlike the Gestalt IT Tech Field day, the EMC Customer Council is governed by a non-disclosure agreement which means I cannot blog about any of the content discussed. However it’s a small price to pay when you get invited to an extremely well organised, well attended event where all parties involved get something out of it.

It’s easy to see why companies are starting to catch on to the benefits of engaging the customer community directly. In some instances the community becomes a self help group of sorts as well as an alternative marketing channel for a vendor. I often see “a community” leading the way with product information awareness, problem resolution, best practice and procurement advice. The VMware community stands as  one of the best examples of this,  there is a wealth of information out there and it’s not hard to find if you ever need to go looking. In fact if you use twitter or subscribe to an RSS feed like PlanetV12n more often than not the information lands in your lap without you needing to ever look for it.

I wanted to briefly cover off the Gestalt IT tech day. Stephen Foskett the organiser and chief recently set out on a mission to organise a technical field day that vendors would sponsor without the usual NDA’s being in place. Thus allowing the attending bloggers to write about what they saw until they couldn’t possibly write anymore.  He did an exceptional job and I believe the experience didn’t put him off, he’s already looking at organising Gestalt IT Tech Day 2.

Well the attending bloggers wrote post after post and there was lots of good stuff coming out from the vendor visits they participated in. This event is another good example of vendors engaging successfully with the community and everyone getting something out of it. The vendors get a chance to spread the word about their products and services and the bloggers get lots of technical content to put out there for their readers.  Everyone is a winner and that is exactly what a vendor event should be all about.

To read more about the Gestalt IT Tech day and sample some of the many articles written, click the link. What a Tech Field Day!

General, Gestalt-IT, Storage EMC Customer Council, GestaltIT Tech Field Day

I’ve been chatting to an ex colleague recently who was trying to get a Scottish VMware User Group setup.  Through his hard work and determination he’s finally managed to get it off the ground, here’s hoping that we get enough interest to keep it up and running in the future.

The first meeting agenda looks good with Mike Laverick of RTFM attending to discuss vSphere 4 storage.  We also have a talk from our hosts for the day, State Street alongside EMC about the storage setup for their VMware infrastructure.

If you’re interested in becoming a Scottish VMUG member click the link and sign up

If you want to attend this particular VMUG click the register now button

General, VMware, vSphere VMUG

I was approached some months ago by Stephen Foskett about joining a new site that he was creating called GestalIT.  At the time I was in the middle of a major work project and did not have the time to respond,  however this week I’ve been back in contact with Stephen and I’m now an author on GestaltIT. Over time I will be contributing what I feel are some of my better articles to the GestaltIT site as well as publishing articles on VirtualPro.

If you haven’t been to GestalIT I would highly recommend heading over and checking it out,  especially if storage, cloud computing or virtualisation is your thing.

General

During the recent troubles in Iran a large number of people turned their Twitter avatars green to show their support.  Today I recieved an email about a great new site called called Twibbon which allows you to support your cause by adding an overlay to your Twitter avatar. As you can see from my own Twitter profile pic my particular cause in this case is VMware, so to  join up and support VMware go to  http://twibbon.com/join/VMware

The story behind this site is brilliant,  as I heard it a developer based in Edinburgh, Scotland came up with the idea at some ungodly hour last Wednesday had it live by Thursday and they had over 100,000 hits in the first 30 hours alone.  Great idea,  developed quickly and has great potential for the likes of charity support or product support.

General, VMware Twitter

I recently had a number of vSphere ESX4i  USB Key installs following my article on putting vSphere ESX4i on a USB key / Pen Drive. I needed to format a couple for general windows usage, only to find that the ESX4i image creates a number of partitions on the USB Key. Unfortunately Windows does not appear to support the removal of partitions on removable devices so when I was trying to format a 2GB USB stick I was able to format a 110MB partition and that was it. I was a bit stuck on the best way to rectify the issue and wasn’t finding much to help out on the web.

That’s when I stumbled upon the HP USB Storage Format Tool,  a great little tool that works with a wide range of USB sticks and not just HP ones.  It allowed me to wipe the USB key as a single entity and didn’t care about the partitioning, returning my USB Key to a useable state within windows.

You can download it from HP’s website by clicking on this link,  sometimes you just don’t know if you can trust other download sites.

General ESX 3.5i, esx4i, USB, VSphere

I was discussing the cloud computing concept with a colleague the other day and to be honest,  I was making a bit of a mess of it.  I think I only served to confuse him more.

So I put the call out on twitter to see if someone had any good videos that explained cloud computing in plain English.  Luckily I didn’t have to wait long for a reply,  Simon Long to the rescue with a couple of videos he had hosted on his website the SLOG. Simon is an active member of the virtualisation community whether that is on his blog, twitter or the VMTN community website.  Thanks for these Simon!

General Cloud Computing

I was reading through my google reader this morning on the way to work and noticed the excellent Duncan Epping over at Yellow Bricks blogging about the next incarnation of the Blue Bear Kodiak software.

I still get a lot of interest in the Beta Invites for bluebear (I’ve got 4 left) and the stats show that it’s a popular page.  So I’m sure there are plenty of people out there who’ll be pleased to hear there is a new version on the way.  Check out Matt Millar’s (Papa Bear) video which provides a great overview of the kodiak product and a sneak preview of some of the new features.

Soon as I get my hands on the latest version, I’ll post a review.  If anyone wants one of the few remaining Beta invites which should cover you for the existing and new version,  please just comment on this post.

General bluebear, ESX management

I’m back,  been very busy on a project implementing 2 new EMC CX4’s and Symantec Enterprise Vault so not had a lot of time to write any blog posts.  However I’ve got a fair bit of content to put out there from my Symantec EV implementation which should hopefully help others in the same boat. 

The first one I’d like to write about is Public Folder Journaling.  The company I work for have over 1,500 mail enabled public folders,  the majority of which appear to have been auto-created when we were on Exchange 5.5. The first major problem I faced was the fact that Exchange 2003 does not support the journaling of mail enabled public folders.  For Compliance reasons it was essential to capture these emails so I had to come up with a suitable solution.

I came up with the idea of forwarding all mail enabled public folders to a mailbox, thus allowing it to be captured by the exchange journaling and then sucked up into the Symantec Enterprise Vault product.  But I had the small matter of identifying which public folders were being actively used for email. Then the task of finding them within the public folder hierarchy, on top of that I needed to manually apply the forwarding address setting.

This was going to take weeks to complete! So I decided on a blanket approach and looked at adding the forwarding address to all 1,500 public folders. I figured that if the mailbox was in use then the mail would be captured, if it wasn’t then no damage done. But how exactly do you change the settings of 1,500 public folders?

ADModify to the rescue! For those of you unfamiliar with ADModify it’s a great little tool for doing bulk changes to Active directory objects and their attributes,  you can find more information on Technet here.  In this case I wanted to change two values,  the ones relating to the forwarding mailbox and the check box so that the message would be delivered to both locations.

So I did the following,  used ADModify to bring up a list of all mail enabled public folders.  I used the custom LDAP Query option within ADModify and used the query (&(objectClass=PublicFolder)(Mail=*).

I selected 3 public folder objects to test with before doing all of them of course.  Once selected click next and you will be presented with a screen very similar to normal active directory so you can change one of the many common attributes and AD object has.

So I navigated to the Exchange General Tab and added the full path canonical path to the user object I’d setup with the mailbox, “PFCapture”

The check box attribute was a little trickier as it wasn’t shown in ADModify,  after much searching with ADSI Edit I found it was a Boolean value called deliverandredirect. So I went to the custom tab and entered the attribute name and the value TRUE so it would put a tick in the check box.

I applied these changes and after a little waiting for replication, etc the new attributes appear when examining the objects in ADSI Edit.  Emails sent through to the original addresses found their way to my new PFCapture mailbox and when searching within the Symantec product I can see the emails have been captured for compliance purposes.

So now on to the next problem,  how do I automate the clearing down of the PFCapture mailbox so it doesn’t just grow and grow?  Well Exchange 2003 has a handy feature called mailbox management policies and with this you can control what happens to the messages within mailboxes.

The full instructions on how to set this up can be found in Microsoft Article 319188.  In summary I set up this policy and changed it so any email in any folder within the mailbox was deleted immediately after 1 day.  I then configured the mailbox management process on the server properties (Within Exchange System Manager) to run at midnight each day of the week.  This keeps the mailbox light on space utilisation without any manual intervention,  which is always a good thing.

So there we have it,  a crudely fashioned means of journaling mail enabled public folders. Any questions please just drop me a comment and I’ll get back to you.

General Exchange, Journaling, Symantec Enterprise Vault

I’ve been considering purchasing a server for some time,  so I can setup a home lab for training purposes.  I’ve been reading about the HP Proliant ML110 G5 and ML115 G5 make good ESX hosts in home lab environments.  I had a conversation with Simon over at www.techhead.co.uk recently about how the price of these servers had shot up from a very appealing £120, to well over £350. I was kind of facing up to paying the extra,  but Simon informed me he had a deal brewing regarding these servers and to hold on.

Well he’s now posted the details on how to get these servers for £199, ex VAT and get free delivery thrown in. I’ve personally got my eye on the Quad Core AMD Opteron based ML115 G5,  it only comes with 512MB of memory however I found 4GB for it at a cost of £45.99 from Crucial memory. For another £28 you can add another HP NIC

Get on over to his blog post for further details, take advantage of this deal and get your home lab up and running without delay.  Good Deal & Free Delivery on HP Proliant ML110 & ML115 G5

General esx, Whitebox