VirtualPro

So I haven’t done a lot of real time blogging at VMworld this year as I’ve been busy trying to see and soak up as much as possible.  It’s not every day that you get access to the likes of Chad Sakacc (VP EMC / VMware alliance) Scott Drummond (EMC – ex VMware performance team) and a whole host of other technology movers and shakers. As you can imagine I took full advantage of these opportunities and blogging became a bit of secondary activity this week.

However, I’ve now had time to reflect and one of the most interesting areas I covered this week which was the new storage features in vSphere 4.1. I had the chance to cover these in multiple sessions, see various demo’s and talk about it with the VMware developers and engineers responsible. There are two main features I want to cover in depth as I feel they are important indicators of the direction that storage for VMware is heading.

SIOC – Storage I/O Control

SIOC had been in the pipeline since VMworld 2009, I wrote an article on it previously called VMware DRS for Storage, slightly presumptuous of me at the time but I was only slightly off the mark. For those of you who are not aware of SIOC, to sum it up again at a very high level let’s start with the following statement from VMware themselves.

SIOC provides a dynamic control mechanism for proportional allocation of shared storage resources to VMs running on multiple hosts

Though you have always been able to add disk shares to VM’s on an ESX host, this only applied to that host, it was incapable of taking account of VM I/O Behaviour of other VMs on other hosts. Storage I/O control is different in that it is enabled on the datastore object itself, disk shares can then be assigned per VM inside that datastore. When a pre-defined latency level is exceeded on a VM it begins to throttle I/O based on the shares assigned to each VM.

How does it do this, what is happening in the background here? Well SIOC is aware of the storage array device level queue slots as well as the latency of workloads.  During periods of contention it decides how it can best keep machines below the predefined latency tolerance by manipulating all the ESX Host I/O Queues that affect that datastore.

In the example below you can see that based on disk share value all VM’s should ideally be making the same demands on the storage array device level queue slots.  Without SIOC enabled that does not happen. With SIOC enabled it begins throttling back the use of the second ESX host’s I/O queue from 24 slots to 12 slots, thus equalising the I/O across the hosts.

Paul Manning (Storage Architect - VMware product marketing) indicated during his session that there was a benefit to turning SIOC on and not even amending default share values.  This configuration would immediately introduce an element of I/O fairness across a datastore as shown in the example described above and shown below.

So this functionality is now available in vSphere 4.1 for Enterprise Plus licence holders only.  There are a few immediate caveats to be aware of, it’s only supported with block level storage (FC or ISCSI) so NFS datastores are not supported. It also does not support RDM’s or datastores constructed of extents, it only supports a 1:1 LUN to datastore mapping. I was told that extents can cause issues with how the latency and throughput values are calculated,  which could in turn lead to false positive I/O throttling, as a result they are not supported yet.

It’s a powerful feature which I really like the look of. I personally worry about I/O contention and the lack of control I have over what happens to those important mission critical VM’s when that scenario occurs. The “Noisy Neighbour” element can be dealt with at CPU and Memory level with shares but until now you couldn’t at a storage level. I have previously resorted to purchasing EMC PowerPath/VE to double the downstream I/O available from each host and thus reduce the chances of contention.  I may just rethink that one in future because of SIOC!

Further detailed information can be found in the following VMware technical documents

SIOC – Technical Overview and Deployment Considerations

Managing Performance Variance of applications using SIOC

VMware performance engineering – SIOC Performance Study

VAAI - vStorage API for Array Integration

Shortly before the vSphere 4.1 announcement I listened to an EMC webcast run by Chad Sakacc.  In this webcast he described EMC’s integration with the new vStorage API, specifically around offloading tasks to the array. So what does all this mean, what exactly is being offloaded?

 
So what do these features enable? Let’s take a look at them one by one.

Hardware assisted locking as described above provides improved LUN metadata locking.  This is very important for increasing VM to datastore density.  If we use the example of VDI boot storms, if only the blocks relevant to the VM being powered on are locked then you can have a more VM’s starting per datastore.  The same applies in a dynamic VDI environment where images are being cloned and then spun up; the impact of busy cloning periods, i.e. first thing in the morning is mitigated.

The full copy feature would also have an impact in the dynamic VDI space, cloning of machines taking a fraction of the time as the ESX host is not involved. What I mean by that is when a clone is taken now, the data has to be copied up to the ESX server and then pushed back down to the new VM storage location.  The same occurs when you do a storage vMotion, doing it without VAAI takes up valuable I/O Bandwidth and ESX CPU clock cycles. Offloading this to the array prevents this use of host resource and in tests has resulted in a saving of 99% on I/O traffic and 50% saving on CPU load.

In EMC Labs a test of storage vMotion was carried out with VAAI turned off, it took 2 mins 21 seconds.  The same test was tried again with VAAI enabled, this time the storage vMotion took 27 seconds to complete. That is a 5x improvement, and EMC have indicated that they have had a 10x improvement in some cases. Check out this great video which shows a storage vMotion and the impact on ESX and the underlying array.

There is also a 4th VAAI feature which has been left in the vStorage API but is currently unavailable, Mike Laverick wrote about it here. Its a Thin Provisioning API and Chad Sakacc explained during the group session that its main use is for Thin on Thin storage scenarios. The vStorage API will in the future provide vCenter insight into array level over provisioning as well as the VMware over provisioning.  It will also be used to proactively stun VM’s as opposed to letting them crash as currently happens.

As far as I knew EMC was the only storage vendor offering array compatibility with VAAI. Chad indicated that they are already working on VAAI v2 looking to add additional hardware offload support as well as NFS Support. It would appear that 3Par offer support, so that kind of means HP do to, right? Vaughan Stewart over at NetApp also blogged about their upcoming support of the VAAI, I’m sure all storage vendors will be rushing to make use of this functionality.

Further detailed information can be found at the following locations.

What does VAAI mean to you? – Chad Sakac EMC

EMC VAAI webcast – Chad Sakac EMC

Storage DRS – the future

If you’ve made it this far through the blog post then the fact we are taking about Storage DRS should come as no great surprise.  We’ve talked about managing I/O performance through disk latency monitoring and talked about array offloaded features such as storage vMotion and hardware assisted locking. These features in unison make Storage DRS an achievable reality.

SIOC brings the ability to measure VM latency, thus giving a set of metrics that can be used for storage DRS.  VMware are planning to add capacity to the storage DRS algorithm and then aggregate the two metrics for placement decisions.  This will ensure a storage vMotion of an underperforming VM does not lead to capacity issues and vice versa.

Hardware Assisted Locking in VAAI means we don’t have to be as concerned about the number of VM’s in a datastore, something you have to manage manually at the moment.  This removal of limitation means we can automate better, a storage DRS enabler if you will.

Improved Storage vMotion response due to VAAI hardware offloading means that the impact of storage DRS is minimised at the host level. This is one less thing for the VMware administrator to worry about and hence smoothes the path for storage DRS Adoption.  As you may have seen in the storage vMotion video above the overhead on the backend array also appears to have been reduced, so you’re not just shifting the problem somewhere else.

For more information I suggest checking out the following (VMworld 2010 account needed)

TA7805 – Tech Preview – Storage DRS

Summary

There is so much content to take in across all three of these subjects I feel that I have merely scratched the surface.  What was abundantly clear from the meetings and session I attended at VMworld is that VMware and EMC are working closely to bring us easy storage tiering at the VMware level.  Storage DRS will be used to create graded / tiered data pools at the vCenter level, pools of similar type datastores (RAID, Disk type). Virtual machines will be created in these pools; auto placed and then moved about within that pool of datastores to ensure capacity and performance. 

In my opinion it’s an exciting technology, one I think simplifies life for the VMware administrator but complicates life for the VMware designer. It’s another performance variable to concern yourself with and as I heard someone in the VMworld labs comment “it’s a loaded shotgun for those that don’t know what they’re doing”.  Myself, I’d be happy to use it now that I have taken the time to understand it; hopefully this post has made it a little clearer for you to.

Gestalt-IT, Storage, VMware, vmworld EMC, Storage, VAAI, VSphere

Following my return from my first Tech Field Day I have been reading through my notes and reflecting on the vendors I saw when I was in Seattle.  Of the vendors I saw the one that surprised me most was NEC, everyone has heard of them but not everyone actually knows what they do or what products they make.  As we found out during our visit, NEC have a broad technology portfolio and have quite an interesting offering in the storage space.

Here are some basic facts about NEC that you may / may not know

- Founded in 1899
- Fortune 200 company with over 143,000 staff
- Revenues of $43 Billion in 2009
- $3 Billion spent in R&D each year across 12 R&D global labs
- 48,000 patents worldwide.
- Have been in storage since 1950 

So with that little history lesson over, the main focus of our visit was NEC’s HYDRAstor. This is their modular grid storage offering for customers with backup and archive storage in mind. It’s marketed as “Grid storage for the next 100 years” which may sounds a little far fetched, but data growth and data retention periods are ever increasing.   From what I saw and heard the HYDRAstor could very well live up to this bold claim.

There was a lot of content delivered on the day and the session went on for 4 hours, so I’ve tried to wrap up some of the key features below. I have expanded on the key elements of the HYDRAstor that really caught my attention as I think they are worth exploring in more detail.

Key Features

- 2 tier architecture based entirely on best of breed Intel Xeon 5500 based servers

- 2 tier architecture consists of front end accelerator nodes and back end storage nodes

- Shipped as a turnkey solution, though entry level can be bought for self racking.

- Supports a Maximum of 165 Nodes, 55 accelerator nodes and 110 storage nodes

- All interconnects based on 1GB Ethernet Networking (NEC Network switches included)

- Supports old and new node modules in the same Grid for easy node upgrade and retirement.

- Supports volume presentation with NFS and CIFS (SMB Version 1)

- Non-disruptive auto reallocation of data across any additional grid capacity - DynamicStor

- higher levels of resilience than RAID with a reduced capacity overhead (See DRD below)

- WAN optimised grid to grid replication minimises network bandwidth requirements – RepliGrid

- WORM Support for secure retention / compliance governance - HYDRAlock

- Efficient drive rebuilds, only rebuild the actual data not the whole drive.

- Global inline de-duplication across the entire grid – DataRedux™

- Tight backup vendor integration – strips out backup metadata to improve de-dupe ratios

- Mini HYDRAstor appliance available for remote offices or offsite DR replication.

Data Protection - Distributed Resilient Data™ (DRD)  

The resilience provided by HYDRAstor really caught my eye, primarily because it was so different from anything I had ever seen before.  Distributed Resilient Data (DRD) uses something known as erasure coding to provide extremely high levels of resilience. Now you may think that this would come with a considerable storage and performance overhead, but you’d be wrong.

The HYDRAstor provides 6 levels of protection (1 – 6) all with differing levels of protection and capacity overhead. With the default level 3 selected NEC’s implementation of erasure coding splits the data chunks into 12 parts, 9 data and 3 parity. The use of erasure coding means that it only ever needs 9 parts to make up a complete data chunk. So if that data chunk is spread over 12 disks in a single storage node, it can withstand 3 disk failures. if those 12 chunks are spread over 12 storage nodes then you can withstand 3 complete node failures.

This default level 3 protection requires a 25% capacity overhead, much like RAID 5.  However by providing for 3 disk failures it provides 300% more protection than RAID5 and 150% more protection than RAID 6.  If you want to go to the highest level of protection (level 6) then there is a 50% capacity overhead as with RAID 1, however you can withstand the failure of 6 disks or 6 nodes.

The following video describes Distributed Resilient Data™ (DRD) at the default level 3

 

High Performing

The demonstration NEC gave us was based on their lab setup of 20 accelerator nodes and 40 storage nodes.  This was a 4 rack setup, which as you can see from the photo below is not a small setup. What it is though, is a very high performing storage solution.

NEC demonstrated a data copy that utilised a full 10GB per second throughput, which worked out at about 540MB throughput per front end accelerator node.  The screenshot from the management GUI below shows the  total throughput achieved.

The maximum HYDRAstor configuration consists of 11 racks and is capable of 25GB per second or 90TB per hour. This works out at roughly 2 PB’s in a 24 hour period, that is an astounding amount of data throughput.  Surely a level of throughput to deal with even the most demanding backup or archiving use case.
 

There were a few negative aspects that I picked up on during our visit, thankfully all ones I feel can be addressed by NEC over time.

User Interface

I felt the user interface was a little dated (see screenshot above), it served it’s basic purpose but wasn’t going to win any awards. It was a stark contrast when compared with the very nice and easy to use GUIs we saw from Nimble storage and Compellent.  That said if the HYDRAstor is only being used as a backup and archive storage and not primary storage, does it actually need to have the worlds best GUI, possibly not.

Solution Size

The HYDRAstor came across as a large solution, though I’m not sure why. When I think about it any storage solution that provides 10GB/sec throughput and 480TB of raw storage is likely to take up 4 racks, in some instances probably a lot more.  Maybe it was the sheer number of network interconnects, perhaps some consolidation with 10GB Ethernet could assist in making the solution appear smaller.  NEC could also look at shrinking down the servers sizes, probably only possible with the accelerator node servers as the storage nodes need 12 x 1TB disk so not a lot of scope for size reduction there.

Marketing

A general consensus among delegates was why have NEC marketing not been pushing this harder,  why had so many of us in the room not heard about it? I suppose that was one of the reasons we were there, to hear about it, discuss it and ultimately blog about it as I’m doing now. There are some specific target markets that NEC maybe need to look at for this product, possibly looking at world wide data retention regulations as a means of identifying potential markets and clients.  More noise needs to be made by NEC about there efficient de-dupe integration with enterprise backup products such as CommVault Simpana, Symantec NetBackup, TSM and EMC Networker.  More comments such as the one below wouldn’t hurt.

with the application aware de-duplication for CommVault we’ve optimized storage efficiency with a four times improvement in space reduction.
Pete Chiccino, Chief Information Officer, Bancorp Bank

EMEA availability

NEC told us that this product is not being actively pushed in the EMEA region.  Currently the product is only available for purchase in North America and Japan.  One of the points I made to NEC was that the HYDRAstor appeared to me to be a product that would have a lot of applications in the European market place, possibly more so in the UK.  I made specific reference to FSA regulation changes where Financial companies are now required to keep all electronic communications for up to 7 years.  NEC’s HYDRAstor with it’s high tolerance for failure, global de-duplication across all nodes and grid like extensibility is perfect for storing this kind key critical complaince data.  That is a very specific example, another is insurance companies who have longer retention requirements and museums digitising historical documents / books which have a “keep forever” retention requirement.

NEC contacted me via twitter after the event to say that although not on sale in EMEA if a company has a presence in the US they will be able to explore purchasing the HYDRAstor through NEC America.

Summary

I had no idea what to expect when we arrived at NEC’s offices, sure I knew who they were but I had no idea what they were doing in the storage space. Gideon Senderov at NEC certainly saw to it that we had all the information needed to form an opinion, his knowledge of his product was simply outstanding.

NEC HYDRAstor is a product that is quite unique. It’s easy to scale up and scale out, has high levels of redundancy without the normal capacity penalty and of course exceptional levels of performance. It strikes me as a product that any IT professionals responsible for backup, archiving and long term data retention would be very, very interested in

Note : Tech Field Day is a sponsored event. I receive no direct compensation and take personal leave to attend, however all event expenses are paid by the sponsors via Gestalt IT Media LLC. The views and content expressed here are my own and is in no way influenced by the sponsors of this event.

Events, Gestalt-IT, Storage GestaltIT Tech Field Day, HYDRAstor, NEC

It’s now been a couple of days since the second day of the Gestalt IT Tech Field Day, I’m actually taking the opportunity to write this on the plane on the way back from Seattle. So once again I thought I would do a summary post until I get the chance to write up a detailed post on each vendor.

 

Compellent were one of the main sponsors for the Seattle Tech Field Day and were responsible for us getting access to the Microsoft Campus. So a big thank you to Compellent for their support of Tech Field Day.

Compellent are a company I have had dealings with before, I looked at buying one of their storage devices back in 2008 and was very impressed by the product they had on offer at the time.  This was a great chance for me to revisit Compellent two years on and see how things had changed.

Compellent in general still appears to be much the same product that I liked so much back in 2008.  Their pooled storage model, software controlled RAID write down, space efficient snapshots and WAN optimised thin replication are all superb  features. There main differentiator back in 2008 was their ability to do automated storage tiering (Data Progression™), something that others in the industry are starting to catch up to (EMC FAST). Compellent’s Data Progression technology is one that many customers actively use with good results, I was slightly disappointed though to learn that their data movement engine only executes once every 24 hours and cannot be made more frequent.  I’m not sure how that compares to EMC FAST but is something I’ll include in a more expansive post.

A feature I had heard of but didn’t quite understand previously was Compellent’s Live Volume.  It’s another unique feature for Compellent and one of my fellow delegates even described it as “EMC vPlex that you could actually afford”. Compellent implement the Live Volume feature at software level as opposed to a hardware based implementation like EMC vPlex. Compellent are able to present the same volume, with the same identity in two different locations, they do this using the underlying WAN optimised asynchronous replication. One point of note was that this is not an active / active DR like setup,  this is a setup for use in a controlled maintenance scenario, such as SAN fabric maintenance or a DC Power down test.

Compellent also took the opportunity to share some roadmap information. Highlights included the release of the 64 bit, Series 40 Controller base on the Intel Nehalem, encrypted USB device for seeding replication, a move to smaller 2.5” drives and 256 bit full disk encryption among others.

 
Although we were situated on Microsoft’s Campus for a large part of Tech Field day we were never presented to by Microsoft, which was a shame.  We did however get the chance to visit the Microsoft store which is for employees only.  It gave us all a chance to buy some discounted Microsoft Software and souvenirs of our visit to Redmond which we all took advantage of.

Tech Field Day delegates Kevin Houston, Stephen Foskett and Jason Boche using their iPhones and iPads in the heart of the Microsoft campus. Note Jason Boche using an iPad and wearing his VMware VCDX shirt, brilliant!

Our afternoon session was spent a short bus ride away from Microsoft at NEC America’s Seattle office.  We were here to hear about NEC’s storage offering (I had no idea they even did storage) and more specifically the NEC HYDRAstor range. We had a very in depth session on this fascinating product with Gideon Senderov, Director of Product Management for the HYDRAstor range.

NEC have taken an innovative approach with this product, one I was not expecting. They utilise full blown NEC servers to provide a two tier architecture made up of front end accelerator nodes and back end storage nodes.  On top of this they don’t use the traditional RAID model, instead using something known as erasure coding to provide improved data protection. I will deep-dive this particular data protection method in another article but it was a very interesting and different approach to what I’m used to.

The HYDRAstor grid is marketed as “Storage for the next 100 years” and with it’s grid architecture it’s reasonably easy to see how that statement could be realised.  You can add additional nodes into the grid and it will automatically redistribute itself to take advantage of the capacity.  You can also mark nodes for removal,  the system evacuating the data to enable nodes to be removed from the grid.  This combined with the ability to co-exist old and new HYDRAstor nodes shows why it’s a good storage location for data with a very long term retention requirement.

It appeared to me that HYDRAstor was designed specifically as a location for the output of archive or backup data and not a primary data storage solution. The reason I say this is that when we discussed in-line de-duplication the product was already integrated with major backup vendors (Symantec NetBackup, CommVault Simpana, Tivoli Storage Manager and EMC Networker). NEC were getting very clever by stripping out metadata from these backup vendors to improve the level of de-dedupe that could be achieved with the product when storing backup data.

I will revisit the HYDRAstor, once I have had a chance to go over my notes I fully intend to dedicate a full article to it as I was very impressed.

          

Rodney Haywood and Gideon Senderov white boarding the configuration of the NEC HYDRAstor

Note : Tech Field Day is a sponsored event. I receive no direct compensation and take personal leave to attend, however all event expenses are paid by the sponsors via Gestalt IT Media LLC. The views and content expressed here are my own and is in no way influenced by the sponsors of this event.

Events, Gestalt-IT, Storage GestaltIT Tech Field Day

I have been trying desperately this week to keep up to date with the latest announcements coming out of EMC World 2010.  Problem is they appear to be making them and blogging about them faster than I can read and assimilate them.

One blog post that did catch my attention was a post by EMC’s Chad Sakac. Chad constantly amazes me, he generates a massive amount of super high quality technical content for the EMC and VMware community. His blog post was entitled “EMC’s next generation vCenter Plugins” and details the latest and greatest offerings from EMC’s Free vCenter plugins.

The Virtual Storage Integrator (VSI) V3.0.0 is a renaming of the existing EMC Storage Viewer 2.1 plugin that has been available for a while.  Why the rename? Well EMC are introducing tighter integration by enabling storage provisioning from within vCenter, it’s now surpassed being just a storage viewer.  The storage provisioning integration works with the CLARiiON across all protocols (FC, ISCSI, FCOE) and it also works with NFS on the Celerra. It also adds greater degrees of simplicity and reduces risk by automating all the tasks involved in provisioning and presenting storage to your vSphere cluster.

Chad explains it in much more detail and much better than I ever could in the following video.

I personally feel that the benefits of EMC’s ownership and tight working relationship with vSphere are beginning to shine through.  Such tight levels of integration are now being delivered and future development doesn’t look likely to slow down either. The quote from Chad below show’s how aggressively his team are working to constantly bring new features to the table and best of all, there completely free!

EMC Virtual Storage Integrator (or VSI) is the main EMC vCenter plug-in.  Over time, more and more functions that currently require installing one or more additional plugins will fold into the EMC Virtual Storage Integrator.  We’re beefing up the team behind it, and they have a very aggressive roadmap (wait till you see what’s next!!!)

Click the link below to find out more about what vCenter plugins are available, what they’re designed to do and where you can download them from in EMC Powerlink.

Storage, VMware, vSphere EMC, Plugins, vCenter, VSI, VSphere

One of the things I’ve come to love about blogging is the fact that I occasionally get contacted by the odd tech start-up. Keen to demonstrate their latest market leading idea that is going to revolutionise the industry as we know it.  Earlier this month I was contacted by Mindy Anderson who is the Product director at tech start-up Virsto (short for Virtual Storage). Virsto had a new product for Microsoft Hyper-V that they wanted to demonstrate to me in advance of their big product launch. Having looked at Mindy’s background in the storage industry I was very keen to hear more about their new product.

The product is called Virsto One and is aimed solely at Windows 2008 R2 Hyper-V. The product introduces some new features like thin provisioned clones & snapshots, that expand the functionality of the standard Hyper-V product. The most interesting feature in my opinion though is the attempt to tackle the virtualisation / storage problem commonly known as the I/O blender effect.

So what does Virsto One look like?

The software itself installs in the parent partition of each Hyper-V host and consists of the filter driver, a system service and a VSS provider.  The filter driver sits above the raw storage (any block storage) and presents a VHD object to the parent partition.  This setup allows users to configure physical storage once and then use Virsto One to carry out all future provisioning tasks. This includes full support for creating  thin provisioned, high performing, cluster aware snapshots and clones from either the Virsto One Hyper-V MMC snap-in or Powershell.

So what about the I/O blender effect?

Most storage technologies are not designed for the virtual data centre, most are still designed around the one to one physical server to storage model. Think of a number of virtual machines all with predictable I/O behaviour (if you think of them as physical).  What tends to come out of the physical hypervisor host is a large amount of completely random I/O.  Random I/O has an obvious performance impact when compared with sequential I/O so as you increase the number of VM’s you increase the random I/O from your Hyper-V host.  So as VM density increases performance drops, as we all know low VM density is not your objective when you embark on a virtualisation project.

So Virsto One has an interesting way of dealing with this. Although the “secret sauce” has never been divulged in-depth in its basic form they journal the random I/O that comes down from the Hyper-V host to staging disk.  A staging area is required per physical Hyper-V host and about 20GB / 30GB of disk should support multi-terabyte write downs through use of de-dupe technology. Periodically the data in the staging disks will be flushed / written down to the primary storage location, at this point the Random I/O is laid down sequentially on primary storage to improve read performance. Virsto indicated that in time they would look to support multiple de-stages so that data could be de-staged to another array for business continuity purposes or to the cloud for disaster recovery purposes.

Are there any performance figures to back this up?

Performance figures from the Virsto test lab show the I/O Blender effect in full effect as VM density increases in the standard Microsoft setup.  With the Virsto software sitting in the middle, staging the data and de-staging it sequentially, there is an obvious improvement in performance.  These test results were from Virsto’s own lab and I stressed the importance of having these independently benchmarked by customers or an external consultancy.  Wendy indicated to me that this was something they were looking into,  I look forward to reading and sharing the whitepaper when it is eventually produced.

So who would be interested in a product like this?

Well ideally the product would benefit Hyper-V customers who require high density, high performing virtual environments.  Hosting companies making use of Hyper-V for selling virtual server instances may well see Virsto as a good way of increasing performance and reducing costs through the use of golden images, snapshots, etc.  Who knows though,  individual companies with an investment in Hyper-V may well see the benefit in this kind of product.  In a way I see it’s not to dissimilar to a company buying PowerPath/VE to increase I/O performance in a vSphere environment.

It is important to note that although this product has been initially built for Microsoft Hyper-V the principals behind it are hypervisor agnostic.  I asked the question “why Hyper-V?” at the start of my chat with Virsto,  the answer was that Hyper-V had functionality gaps and was easier to integrate into.  VMware on the other hand is a more mature product where VMFS has gone some way to deal with the traditional virtualisation storage problems.  Citrix virtualisation customers will be happy to hear that testing has already begun in the lab with a version of Virsto one for XenServer, ETA unknown at this stage.

So how much does all this cost?

At the time of the interview,  which was a good few weeks back the per socket price being talked about was $1,000 - $2,000 USD per socket, again not to dissimilar to the pricing for EMC PowerPath/VE.

Conclusion?

My impression at the time of the demo and interview was that this was an interesting product and very clever idea. The main selling point for me was the increase in performance, if it can be independently verified you would think the product will simply sell itself.  I look forward to hearing more about Virsto in the future and I am particularly interested to see what they can do for other hypervisors especially VMware vSphere with it’s new storage API’s.

Hyper-V, New Products, Storage hyper-v, Storage, Virsto

Over the last month or so I’ve had two invites to participate in vendor events abroad.  The first was an invite to the Gestalt IT tech day in San Francisco, the second was an invite to the EMC EMEA Customer Council event in Prague.  Now as much as I would love to go to everything I get invited to, I have a day job which pays the bills so in this instance I had to chose the one most relevant to my employer and that was the EMC EMEA Customer Council.

Having never been invited to an EMC Customer Council event before I wasn’t entirely sure what to expect. The basic structure of the event involved EMC sharing product roadmap and strategy, deep diving a few key technologies / strategies and then listening to customer feedback.  The sessions I attended were very interactive round table discussions, with a lot of enterprise customers who were not backward in coming forward with their feelings and opinions. As the sessions went on I started to see why EMC run these events. It would be hard to gain this kind of candid and honest feedback through any other medium, this kind of information is invaluable to a vendor. From my perspective as a customer I got a lot of good insight into roadmap, allowing me to more accurately propose a long term EMC storage strategy for my employer.  I also got to meet and chat to a lot of interesting people and best of all, I got to hear about the experiences of other customers. It was re-assuring to hear that whether you are an SMB IT operation or an enterprise level one, you tend to have very similar issues. The only difference sometimes being the scale of the infrastructure involved.

Now unfortunately unlike the Gestalt IT Tech Field day, the EMC Customer Council is governed by a non-disclosure agreement which means I cannot blog about any of the content discussed. However it’s a small price to pay when you get invited to an extremely well organised, well attended event where all parties involved get something out of it.

It’s easy to see why companies are starting to catch on to the benefits of engaging the customer community directly. In some instances the community becomes a self help group of sorts as well as an alternative marketing channel for a vendor. I often see “a community” leading the way with product information awareness, problem resolution, best practice and procurement advice. The VMware community stands as  one of the best examples of this,  there is a wealth of information out there and it’s not hard to find if you ever need to go looking. In fact if you use twitter or subscribe to an RSS feed like PlanetV12n more often than not the information lands in your lap without you needing to ever look for it.

I wanted to briefly cover off the Gestalt IT tech day. Stephen Foskett the organiser and chief recently set out on a mission to organise a technical field day that vendors would sponsor without the usual NDA’s being in place. Thus allowing the attending bloggers to write about what they saw until they couldn’t possibly write anymore.  He did an exceptional job and I believe the experience didn’t put him off, he’s already looking at organising Gestalt IT Tech Day 2.

Well the attending bloggers wrote post after post and there was lots of good stuff coming out from the vendor visits they participated in. This event is another good example of vendors engaging successfully with the community and everyone getting something out of it. The vendors get a chance to spread the word about their products and services and the bloggers get lots of technical content to put out there for their readers.  Everyone is a winner and that is exactly what a vendor event should be all about.

To read more about the Gestalt IT Tech day and sample some of the many articles written, click the link. What a Tech Field Day!

General, Gestalt-IT, Storage EMC Customer Council, GestaltIT Tech Field Day

I’m currently revising for an EMC exam and part of the course material involves doing some training on the Navisphere simulator. No problems, except where on earth do you download the simulator software? I searched for ages before finding out that you only get access to it on EMC PowerLink if you are an EMC employee, partner or you have bought the relevant training.  I had done the latter but still couldn’t get it downloaded for some reason,  that’s when a google search turned up a link to the following page which required no credentials, EMC Education Services product simulators.  ** SEE UPDATE BELOW **

I managed to download the latest version for EMC Flare release 28 without issue.  I was however asked for my EMC PowerLink credentials when I first opened the simulator so perhaps that’s how it’s authenticated against the training purchased. It’s a really good learning aid and I’ve been able to find out how to do things on it that I wouldn’t have dared to attempt on our actual CLARiiON implementation. Hope this link helps someone who’s searching for this,  took me ages to find it.

**UPDATE**

It would appear that the link I had included in the original article is no longer working. Thanks to Mike for commenting on the post to alert me to the issue.  I’ve had a search around and Bas Raayman, one of my fellow GestaltIT authors has also written an article on the subject.  He has come up with an alternative means of downloading the simulators.

Basically go to the EMC Information and Storage Management web site by clicking the link.  Next click on one of the learning aids links,  you will now need to fill in some survey information but it’s a small price to pay to get free access to the simulators for both Navisphere, CLARiiON CX Series and Centera. Once the survey has been completed you can download various learning resources including the above mentioned simulators.  The registration step still exists when you first log on to the simulator,  you will at the very least require an EMC Powerlink logon to complete this step.

Certifications, Storage Certification, EMC, Navisphere

I’ve recently been evaluating some of the new features in VMware vSphere to see what use they would be to my current employer. One of the areas that I touched upon in my “what’s new in vSphere Storage”  blog post was thin provisioning.  I wanted to come back and cover this particular topic in more detail as it’s a key feature and it’s available throughout all versions of vSphere so I’m sure everyone will be interested in it.

What is Thin Provisioning?

Thin Provisioning in it’s simplest form is only using the disk space you need.  Traditionally with virtual machines if you create a 500GB virtual disk it will use 500GB of your VMFS datastore. With Thin Provisioning you can create a 500GB virtual disk, but if only 100GB is in use only 100GB of your VMFS datastore will be utilised. Credit to Chad Sakac for the diagrams below.

How does it work in vSphere?

Thin Provisioning is being heralded as something new with vSphere,  when in truth it was already available in VI3.  In VI3 creating a thin provisioned disk involved using vmkfstools and was also not a production supported VM configuration.  Now in vSphere the creation of thin provisioned disks can be carried out from the VI Client (see below) and is a supported production configuration for a VM.

It’s as simple as checking a check box, the results are pretty good to.  Below you can see I have created two thin provisioned VM’s on my new ESX4i host and you can see the provisioned space and the used space stats being shown in the VI Client. 

What are the benefits?

The thin provisioning  feature is perfect for my home lab environment where disk space is at a premium, but how does it translate into real world implementations of ESX.  Well I for one have been looking at exactly this to identify what benefits could be achieved within my employers ESX estate. A quick audit found that our development and system test ESX environment was running at 48% disk utilisation,  so straight away thin provisioning would save us 52% on storage capacity used. Paul Manning of VMware mentioned on a recent communities podcast that on average vSphere would save users 50% on storage.   This is possibly not such a big thing when your talking about test environments, but when you move up to production SAN Storage, saving 50% on an expensive SAN array is a very real and tangabile cost saving.  One that people should definately take into account when making a cost benefit case for buying or upgrading to vSphere.

What are the potential downsides?

One of my personal concerns with thin provisioning is the potential overhead on any write activity that would requires the extension of the VMDK file.  To me there is an obvious VMFS operation that needs to take place there which would add to the overall time to complete the disk write.  When there is a requirement to expand a disk, the VMDK files will increase in increments based on the block size of the underlying VMFS partition, 1MB, 2MB, 4MB or 8MB.  So the overhead may be smaller if your VMFS has been formatted with a bigger block size, i.e. for a 16MB write it only has to expand 2 blocks when the VMFS block size is 8MB but would have to expand 16 blocks if it was formatted with the 1MB block size.  I can imagine this percieved overhead could put people off using thin provisioned disks for certain production based environments, especially those where there is a lot of write I/O activity,  SQL Server or Exchange for example.  To counteract that though,  the improvements in the VMware I/O Stack should compensate for this performance overhead.  This could potentially leave you in a situation where you’ve reduced a VM’s storage footprint and still have performance equal to that experienced in VI3,  possibly not a bad trade off.  I’d also expect people running their VMware environment on enterprise SAN technologies from the likes of EMC or NetApp to notice minimal performance impact with thin provisioning as SAN memory caches help take up the strain.

Another downside is if you want to use VMware Fault Tolerance to protect a VM then you cannot use thin provisioned disks.  To be honest this is a small issue as Fault Tolerance protection is most likely going to be on virtual machines that are important to your organisation.  These machines are probably the ones you wouldn’t thin provision in the first place for performance reasons.

Thin provisioning creates it’s own unique problem in that what we’re basically doing here is over provisioning the storage.  You need to keep a very close eye on thin provisioning as it’s quite feasabile that your VMFS datastore could fill up and your virtual machines fall over.  Not what you want to come into on a Monday morning,  or any morning for that matter.  So you need to monitor your storage and ensure that there is enough free space.  One of the simplest ways to do this is through the use of the new alarms in vSphere that allow you to alert on datastore usage and datastore over provisioning.  These should keep you from filling a datastore and killing your VM’s or ESX Servers

One gotcha that you should watch out for is VM swap files, as these are usually stored with your virtual machines vmdk files in the VMFS datastore.  In VI3 the swap file was not deleted when a VM was powered down,  in vSphere the swap file is deleted on power down and recreated when the VM is powered up.  You should be aware of this when over provisioning storage as you could get into a situation whereby you find you can’t power on a VM because there isn’t enough space for the swap file to be created.  This becomes more likely as servers and VM configuration maximum’s increase,  if you have a VM with 20GB of RAM it’s going to need 20GB of disk space for the swap file.  if you have 256GB of RAM in your vSphere host and you allocate it all out to VM’s then you need to think about the 256GB of disk capacity required to service virtual machine swap files.

Storage vMotion

If you’ve already got a VI3 environment then the chances are that your VM’s aren’t thin provisioned,  how on earth are you going to take advantage of this new feature? Well if you have purchased a vSphere edition that supports storage vMotion then you can of course migrate the underlying storage and have it thin provisioned during the move.  This should allow existing VI3 customers to claim back a lot of space,  as I mentioned before I found that our development and test VI environments were only 48% utilised.  If I storage vmotion all those VM’s and thin provision at the same time I will free up about 1.5TB of storage that wasn’t being used in the first place.

I’ve included a video below which demonstrates the Storage vMotion and thin provisoning features in vSphere quite nicely, enjoy!

Gestalt-IT, Storage, VMware, vSphere Storage vMotion, Thin Provisioning, VSphere

This weekend I finally had the chance to catchup on some of the new storage features released as part of vSphere 4.0,  there are quite a few changes to cover,  some of them quite exciting.

VMFS Upgrade

Once of the good pieces of news to come out is that the VMFS changes in vSphere are minimal.  vSphere 4.0 introduces a minor point release (3.3.0 to 3.3.1) with some subtle changes,  so much so that it’s not really been documented anywhere.  Most of the changes with VMFS are actually delivered within the VMFS driver at the VMKernel level,  this is where most of the I/O improvements and features such as thin provisioning have been delivered as part of vSphere.

Upgrading VMFS was a major step in the upgrade from VMFS 2 to VMFS 3,  good to hear that there are no major drivers to upgrade VMFS as part of your vSphere upgrade.  Any new VMFS datastores created with the new vSphere hosts will of course be VMFS 3.3.1 however this is backwardly compatible with earlier versions of ESX 3.x.  If you really want to move onto the new version of VNFS, format some new datastores and use Storage vMotion to move your VM’s onto the new VMFS 3.3.1 datastores. 

Thin Provisioning

Thin provisioning is one of the areas that excites me most about the new vSphere release.  I conducted a very quick survey of my employers development and system test ESX environments recently and found that currently we were only utilising 48% of virtual storage that had been provisioned.  It’s easy to see where immediate savings can be made simply by implementing vSphere and thin provisioning.  I’ll be using that in the cost benefits case for sure!

Thin provisioning is nothing new,  it has been available at the array level for a while now, so one of the big questions is where should I thin provision?  Well that really depends what kind of environment you have I suppose.  Smaller customers will benefit greatly from VMware thin provisioning as they probably don’t own arrays capable of TP.  Bigger companies on the other hand might well benefit from carrying out both as they have both the skill sets and the equipment to full utilise it at both levels. 

Chad Sakac has written a superb article entitled “thin on thin where should you do thin provisioning vsphere 4.0 or array level” which goes deep into the new thin provisioning features and the discussions around what’s the best approach. I strongly suggest people give it a read,  it explains pretty much all you need to know.

Storage VMotion

The Storage vMotion in ESX 3.5 had a few limitations which vSphere addresses.  It’s now fully integrated with vCenter as opposed to being command line based in the previous version,  it allows for the moving of a VM between different storage types, i.e. FC, ISCSI or NFS.  One excellent usage of Storage vMotion is the ability to migrate your thick vm’s and change them to thin VM’s.  Perfect for reclaiming disk space and increasing utilisation without downtime, brilliant!
 
Storage vMotion has also been enhanced from an operational perspective. Previously storage vmotion involved taking a snapshot of a disk,  copying the parent disk to it’s new location and then taking the child snapshot and re-parenting the child disk with the parent.  This process required the 2 x the CPU and memory of the VM being migrated in order to ensure zero downtime.  In vSphere 4.0 Storage vMotion uses change block tracking and a process very similar to how vMotion deals with moving active memory between hosts.  The new Storage vMotion conducts an iterative process scanning what blocks have been changed, each iterative scan should result in smaller and smaller increments and when it gets down to a small enough size it conducts a very quick suspend / resume operation as opposed to using the doubling up resources method that it previously needed to.  Making it faster and more efficient than it was in it’s previous incarnation.

Para Virtualised SCSI

Para Virtualised SCSI (PVSCSI) is a new driver for I/O intensive virtual machines. VMware compare this to the vmxnet adapter,  which is an enhanced and optimised network driver providing higher performance.  PVSCSI is similar, it’s a specific driver that offers higher I/O throughput, lower latency and lower CPU utilisation within virtual machines. Figures discussed by Paul Manning on the recent Vmware community podcast included 92% increase in IOPS throughput and 40% decrease in latency when compared to the standard LSI / BUSLogic virtual driver.

A caveat of this technology is that the guest OS still has to boot from a non PVSCSI adapter (LSI / Buslogic),  you would look to add your PVSCSI adapter for your additional data virtual disks.  Currently only Windows 2003, Windows 2008 and RH Linux 5 have the software drivers to take adavantage of this new adapter.

Update  - Chad Sakac has posted a new EMCWorld I/O Performance comparison of the vSphere PVSCSI adpater vs the LSI SCSI adapter, check out the link for more details.

VMware Storage Book

Paul Manning mentioned on the recent podcast that VMware are planning a book dedicated to Virtualisation and storage in an attempt to consolidate the amount of documentation out there on Storage configuration and best practice.  Currently users need to look through 600 pages of the SAN Config guide and vendor guidelines. VMware would hope to try boil this down to a much more manageable 100 - 150 pages.

If you can’t wait that long, Chad Sakac has written the storage chapter in Scott Lowe’s new vSphere book which I believe is available for pre-order on Amazon

vSphere Storage WhitePaper

Paul Manning who I’ve mentioned in this blog post has written a great 10 page white paper explaining all of these features in more detail along with some of the more experimental features I haven’t mentioned. 

http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereStorage_P10_R1.pdf

Gestalt-IT, New Products, Storage, VMware, vSphere esx, Storage, Storage vMotion, Thin Provisioning, VSphere

Was reading through my RSS feeds this morning while standing in line for a coffee (I love the iPhone) and came across Eric Sloof’s article on Compellent.

Compellent have introduced a powershell based command set to compliment their storage offering.  Now offering the ability to script a lot of the everyday CLI based tasks in a format that is becoming more and more popular.  For me the main benefit I can see is from an integration angle,  linking in windows and storage tasks together is the kind of thing that I can see becoming handy for BCP and DR scripting.

I had a small smile on my face when reading Eric’s article and I will tell you why,  I’ve recently been looking at Compellent for a solution I was building.  Compellent really impressed me with their offering, SAN simplified as far as I was concerned.  The fact that they started from scratch 4 years ago and heavily utilise software management, means that they’re not limited in the same way as some of the other legacy based vendors.  We didn’t end up choosing Compellent for the solution, however the choice was not based purely on the technical angle.  Compellent are still quite small in the UK but do seem to be making serious inroads into the market.  The awards they picked up this year are very impressive and I will be keeping an eye on their growth in the future.

The introduction of powershell to their offering I think shows what a good dynamic company they are. Be interested in any comments anyone has on working with Compellent or their solutions.

New Products, Storage, VI Toolkit / Powershell Compellent, SAN, Storage