VirtualPro

Perhaps the title to this blog post should actually read Docker, Docker, Docker… After all when anyone mentions container technology these days the immediate image that comes to mind is that smiley docker whale!

However I would argue that it should just be containers, containers, containers… The reason I say that is I feel the ecosystem around containers looks significantly broader than just Docker and appears to be diversifying on a weekly basis.  It would be silly however not to acknowledge the fact that the current landscape is dominated by Docker,  as it is the most commonly recognised implementation of containers today. In fact as I go through this article,  you’ll find it ironic that most of the examples I use to explain containers are in fact Docker related, but bear with me I’ll get to the point eventually.

What are containers?

I’m conscious that not everyone reading this will be aware of what a container is… Where have you been for the last year! It’s worth setting out some of the basics and briefly comparing it with a subject people do understand well,  virtualisation.  So how does it differ?  I’ve tried to give a very quick overview below.

 
In simple terms,  containers are lighter in weight and have less memory and compute overhead than virtual machines, they make it easy to support applications that consist of hundreds or thousands of small, isolated moving parts, think of the trend we’re seeing around Micro services here.  A properly containerised application is easy to scale and maintain, so it’s understandable why this would appeal to both developers and operations teams alike.

Where did they come from? / Where are they going?

Containers themselves are not a new invention,  in fact they have been around for years,  consider AIX workload partitions and Solaris Containers alongside todays use of Linux Container technology (LXC) and cGroups.  However it’s thanks to Docker that containers have now become mainstream in the last year. The Docker Engine makes it easy to consume with its simple abstraction,  with the ingenious idea of having pre-built, transportable containers in the form of multi-layered Docker images and a registry mechanism for sharing these images with the world via the Docker Hub.

There are of course other approaches to containers appearing, such as Cannonical LXD and FlockPort both building on top of the native Linux LXC technology. Just last week we’ve also seen CoreOS cause a bit of a storm by deviating away from the Docker path they’ve been treading for a while and setting out to create it’s own container technology called Rocket.

Following their last round of VC funding, it appears that Docker are now moving up the stack and looking to create more of a platform.  With the introduction of Docker Machine (remote management), Docker Swarm (clustering) and Docker Compose (Multi-App configs) Docker has signalled it’s intent to consolidate it’s position as the container technology of choice with smarts wrapped around it.  In my opinion this is very similar to VMware’s move from just selling ESX,  to selling an SDDC platform with all the bells and whistles around automation, monitoring, SDN, SDS, etc.

It’s interesting however that CoreOS feel that container technology still has some fundamental work required (improve security, tooling and openness) and through Rocket are looking to build on the work that Docker have already done .  As CoreOS is a very popular choice for running Docker Containers, I think it’s a great example of coopetition in action.  I also read with interest that Pivotal have expressed a serious interest in getting fully involved in collaborating with CoreOS on the next generation of open containers. Commoditisation of containers anyone!?

Additional Reading

For those that want to dig deeper and do some additional reading on container technology, I would suggest having a little look at the following links for the basics.

What is Docker? – https://www.docker.com/whatisdocker/

Intro to Docker Slide Deck – http://www.slideshare.net/dotCloud/intro-docker-october-2013

Understanding Docker – https://docs.docker.com/introduction/understanding-docker/

Flockport – LXC vs Docker – http://www.flockport.com/lxc-vs-docker/

Wider EcoSystem Momentum

One of the other reasons I think it’s Container, Container, Container and not just Docker is the sheer movement in the wider eco system over the last month alone.

I’ve mentioned the various flavours of containers springing up out there (CoreOS, FlockPort, LXD, etc). On top of that there have been a number of interesting announcements in the platform and orchestration space.

Google announced their Google Container Engine Alpha release, managed by Google’s open source Kubernetes orchestration engine and supporting Docker containers. Google are definitely one of the  the worlds biggest consumer of containers,  spinning up a claimed 2 billion containers per week, now they’re getting into the Container as a Service game. It’ll be interesting to see how they broaden support for other container providers like CoreOS both within the platform and within Kubernetes as an orchestration tool. https://cloud.google.com/container-engine/

Amazon are obviously keen not to miss out on the buzz around containers, at AWS reinvent they launched their EC2 Container Service with support for Docker. Again though Docker is mentioned specifically day 1,  in theory this service offering again could potentially be expanded to interact with other container technologies.  Again it will be interesting to see what Amazon do next to in this space to broaden support and build their CaaS offering  – http://aws.amazon.com/blogs/aws/cloud-container-management/

Microsoft announced support for Linux Containers on Azure back in June 2014 (20% of Azure workloads are now running Linux, can you believe that!). Microsoft have since announced plans to support Docker on top of Windows Server and integrate the Docker Hub with the Azure Gallery service catalogue. So interesting moves afoot from all the big cloud players in the market – http://azure.microsoft.com/blog/2014/10/15/new-windows-server-containers-and-azure-support-for-docker/

All in all the ecosystem looks to be making healthy progress, the big players definitely appear to be  getting caught up in the wave of container mania focused around Docker primarily.  The key thing for me is the openness around orchestration engines like Kubernetes, which will ensure that these tools don’t just apply to Docker but will be capable of embracing new flavours of containers as they appear.

Who is using containers?

I’ve heard of lots of examples, Google, NetFlix, Twitter, Hailo and many, many more.  However as I work for EMC and have an interest in Pivotal so thought I would concentrate on those specific examples. This blog post started following a Docker meet-up in Edinburgh a while back and I was asked if we at EMC were using Docker in production, my answer was…

“well yes sort of, we are using various container technologies within products at EMC”,

The guy looked visibly shocked at this point.  He told me he didn’t know anyone taking that kind of approach to using containers in production and had only ever seen containers used in the realms of application development and testing. In EMC’s case it’s about changing up how we deliver software on commodity hardware and drive our Software Defined Storage strategy.

The above might actually come as a surprise to some,  however at EMC and Pivotal we’re actively adopting and using container technology to change both how we deliver data services and Platform as a Service (PaaS).  In EMC we actively use Linux based containers within the VNXe storage array and our Elastic Cloud Storage (ECS) product, my global boss Chad Sakac alluded to this some time back when the VNXe was launched.

VNXe / Project Liberty

The release of the VNXe was quite key for EMC and the ongoing future VNX array architecture.  This was the first time we had managed to consolidate the VNX file and Block code base on a single x86 platform. Previously the file part of a VNX ran on separate hardware (Data Movers with X-Blades) in front of the block part of a VNX (x86 based storage processors).

Following “Project Liberty” those code bases were architected onto a single linux based kernel.  Now the VNXe runs both the File and Block code on the storage processors (i.e. an x86 commodity server) without the need for dedicated X-Blades in front of the block part of the VNX. We’re still using FLARE and DART code bases to provide the data services,  however in this instance they are being run within CSX containers,  these containers are part of EMC’s C4 Code Base being used for VNXe arrays,  the picture below gives some insight into that architecture where you can see Data Path code being run on the CSX containers.  This is a great example of successful utilisation of containers to deliver simplicity, for EMC internally around how we develop code and build products and for EMC customers around simpler consumption of our data services on VNX.

We’re now seeing this same concept being applied in the new VMAX 3 which boasts our HyperMax OS.  Just last week we’ve launched the latest version of the Enginuity code and with that we can now run the VNX file code as Embedded NAS on the VMAX 3 instead of having to put a physical VNX gateway in front of it.  This is a direct result of the work done to abstract the data services in the VNX,  freeing it from the confines of it’s dedicated X-blade hardware.  I’m not 100% sure what the secret sauce is behind this implementation in the HyperMax OS but in one of his most recent blog posts Chad alluded to it it being similar to what we do in VNXe,  I’m reading that as C4 & CSX

Credit for the most of above goes to Chad who blogged about these things here (Project Liberty),  here (C4 / CSX Containers) and here (VMAX 3)

Elastic Cloud Storage

For those not familiar with EMC’s Elastic Cloud Storage (ECS) it’s a combination of commodity servers, ethernet networking, disk shelves, disks and EMC ViPR advanced data services.  The ViPR data services allow us to layer a mixed combinations of Block, Object and HDFS on top of that same commodity hardware and to do so at huge scale.   These advanced data services of course need to run somewhere,  within ECS this is on top of x86 Intel based servers known within EMC as “Phoenix”,  the number of actual servers / racks depends on your configuration,  an example 1 rack config can be seen below with the Phoenix servers deployed.

ViPR itself is a distributed scale-out platform, ECS utilises ViPR on Docker at the core of the product. The components of ViPR such as the ViPR Controller, ViPR Object storage engine and ViPR Block services are basically individual scale out applications.  Therefore Docker makes a lot of sense as a means of deploying them within the ECS appliance.

In the example below we’re seeing ScaleIO, which is used to deliver the block data services of the ECS appliance.  Running a docker ps –a command returns the current containers running on the docker engine, running on a single Linux host, running on the phoenix hardware.  As you can see there are components of ScaleIO as well as management components such as zookeeper, the ECS fabric and ECS registry.

From a higher level architecture perspective,  what does this look like?  Well below you can see we have 3 ECS appliances operating in a fabric,  with each Linux server running multiple containers and therefore running multiple applications.  So again,  great use of containers to build out a scale out architecture which can deliver exabytes of storage.

Credit for this info & pics go to my EMC Colleague Michael Wehle – ECS Appliance Blog Post

Pivotal use of Containers

So I best not forget the guys over at Pivotal,  who are also using containers within the Pivotal Cloud Foundry PaaS platform.  Today in Pivotal CF 1.3.0.0 containers are used within what are known as Droplet Execution Agents or DEA’s for short,  this is where application instances run within Pivotal CF.  These DEA’s within Pivotal CF are based on an Ubuntu Linux distribution and these DEA’s run what are known as Warden Containers.  The container technology is based on Linux containers utilising namespaces and cgroups for isolation.  Warden itself is a service for managing these containers and determining how clients interact with those containers, i.e. routing network traffic to an app within a container.

Warden is currently being re-written in the language Go and will go by the name Garden,  this is being done as part of Project Diego.  Project Diego is a re-write of the Cloud Foundry elastic run-time and is designed to facilitate easier development changes to the run-time while also allowing it to be more platform agnostic, i.e. support for execution endpoints such as Docker, Linux Containers, Windows and whatever else is just round the corner.

For those interested in the current implementation of Warden containers and how Pivotal are changing this to be more platform agnostic the I recommend watching the following video.  It explains the vision behind re-writing the CF run-time and how it will support that wider ecosystem of endpoints including containers,  pretty cool stuff if you ask me!

Summary

If I very quickly summarise my thoughts after writing this article (which has evolved over a number of weeks) they would be as follows

• Containers are becoming increasingly popular for running ‘12 factor apps’
• There is a healthy container ecosystem with good buy-in from the big tech firms.
• There is a great collaborative open source community bought into containers.
• The Service Orientated / Micro-service approach plays well with container technology.
• Tech companies are already finding novel uses for containers,  i.e EMC and Pivotal.
• Netflix, Twitter and Google are using Containers, effectively, at scale and in production!

I also saw the following tweet recently, which resonated quite well with me as ultimately its about choice and again it’s about mixing and matching to meet your needs.  Containers on VM’s seems a logical implementation to me and in fact is how I’ve always viewed deployment of Pivotal CF containers,  warden containers on DEA virtual machines.

So what next for containers?  I’m really interested to see how the different flavours of containers evolve,  the CoreOS Rocket announcement I mentioned earlier caused some big waves in the industry but makes sense to me.  After that I think is how the various toolsets in evolve to account for these new container technologies and not just Docker, how they’re adopted into CaaS offerings will drive container adoption.  The fact that widespread production based use of containers appears to still be in the propeller head realm (thinking serious DevOp shops + the likes of Netflix) means that how ISV’s embrace and utilise containers as they have virtualisation will also be very important to widespread adoption.

To give an EMC software example, I attended a meeting last week  on EMC’s Captiva, XCP and Documentum products,  these are products related to image capture, business workflow and records management.  They are made up of multiple traditional application components all integrating together to provide different capabilities. After writing this article, reading a lot of stuff around loosely coupled service architectures and looking at what NetFlix have been releasing to the Open Source community all I could think was the following

• Re-develop these components as many smaller micro-services (i.e. 12 factor apps)
• Run them in containers
• Run them within Pivotal CF
• Develop and make changes quickly
• Scale up and down quickly and on demand
• Subscribe to external services that aid the apps, i.e. messaging and data services
• RabbitMQ, GemFire, Hadoop, Object Stores (ECS) to name but a few.

Sounds simple doesn’t it,  be interesting to see how software evolves to embrace these concepts, both within ISV’s and EMC alike!

EMC, General CaaS, Containers, Docker

I’ve had a number of discussions recently around the virtualisation of Hadoop,  some of it with customers,  some of it internal and some of it with my colleagues at VMware and Pivotal.  As always these conversations get me thinking and that in turn has spurred me on to come out of hiding and write my first blog post in a while

This train of thought was sparked by a conversation with someone who was looking at introducing Hadoop into their environment. Primary use case was so they could leverage unstructured data alongside traditional legacy EDW solutions. The conversation centred around the desire to virtualise Hadoop to facilitate quicker provisioning for POC / Test purposes, however there was also a desire to potentially do the same with a production Hadoop environment from a cost perspective.  Sounds pretty straightforward if you ask me,  however the main fly in the ointment appears to be the reluctance from some of the Hadoop vendors to give their blessing or indeed commit to supporting a virtualised approach.

So, why are Hadoop vendors so reluctant?

There could be many reasons why a Hadoop vendor would be reluctant to embrace virtualisation or indeed any abstraction (HDFS from an appliance or Linux containers).  It could simply be a case of them wanting to apply the “keep it simple stupid” approach and ensuring success by building it the way it always has in the past. Or it could be that they’ve simply just never tested it in a virtual environment and unknown is bad in their eyes. Perhaps it’s something else I haven’t even thought of!?

I have my own opinion of course,  I firmly believe that reluctance to virtualise Hadoop is because it fundamentally disrupts today’s Hadoop licensing models and as a result has a negative impact on the vendors revenue streams (per CPU or per host licensing).  This is in essence the same kind of pushback that EMC were getting previously around supporting Hadoop with EMC’s Isilon scale out NAS (per host or per TB licensing).

Let me give you a quick example on how both disrupt the Hadoop vendor.

Say I have 100 commodity servers in my Hadoop cluster,  I have built a Hadoop cluster of 100 servers to provide my required 0.5PB of storage for HDFS (e.g. 5TB per server = 0.5PB).  Now despite having 100 servers I actually only utilise approximately 20% of those servers for Hadoop compute tasks like Map Reduce,  if I want to add more storage I have to add more servers including compute even though I’m never going to use that extra compute power. This model doesn’t fundamentally doesn’t scale well from a data centre footprint point of view.

So first disruption,  lets say I was to separate the compute and storage elements of Hadoop.  I introduce 0.5PB of EMC Isilon to provide the HDFS layer.  Instantly I have reduced my server footprint requirement from 100 servers to 20 physical servers + the EMC Isilon for the storage requirement.  First major advantage,  if I need to scale either compute or hardware I can do it independently of the other. Second major advantage,  I go from 33% capacity efficiency (Hadoop does 3 copies of ever file) to up to 80% capacity efficiency (n+1,2,3 or 4 protection)

Second disruption, now lets say I also introduce virtualisation of the Hadoop compute nodes into the solution. If I was to achieve a VM to physical server ratio of 2:1,  that means I now only need 10 physical servers + EMC Isilon HDFS storage. If I was to go for a more aggressive 4:1 VM to physical server ratio I’d only need 5 physical servers + EMC Isilon HDFS storage instead of the original 100 physical servers.  I’ve now got the ability to easily scale my compute and storage layers independently of each other and have vastly shrunk my data centre footprint, which is a huge cost saving in itself.

Theoretically, we’ve just shaved between 90 – 95 servers off the original solution,  not only have we saved on physical footprint but we’ve also taken 90 – 95 servers worth of annual licensing away from a Hadoop vendor!!  I can see why they wouldn’t be 100% happy about a customer implementing virtualisation and EMC Isilon for Hadoop.

now granted, the above is a very simplistic example.  However if you’re looking at deploying Hadoop I’m sure you can see how virtualisation and introducing Isilon for HDFS could help you massively reduce footprint in the data centre and save on licensing costs. You of course need to build out a suitable TCO analysis for the various options,  however I encourage you to do so as I bet it works out to be quite favourable compared to the original 100 nodes.

What about performance?

One of the other main areas the Hadoop vendors appear to be focusing their FUD (FUD = Fear, Uncertainty and Doubt) is virtualisation and the performance of Hadoop.  Now I don’t disagree that depending on what you’re doing with Hadoop, performance may well be the key requirement.  If it is very, very important to you that your Hadoop is performant and that your Hadoop vendor will support it without argument; then physical Hadoop compute nodes may be the best way for you to go.

That said,  if you deem flexibility and ease of scaling Hadoop (storage or compute) as your key requirement then a different approach may be needed. This is where the separation of compute and storage layers adds immediate benefit and where virtualisation of the Hadoop compute nodes drives increased flexibility.

I should add that I personally don’t think you necessarily have to sacrifice compute performance to gain that increased flexibility.  I would highly recommend reading the following VMware white paper on Virtualised Hadoop Performance with vSphere 5.1. In it VMware conduct a very in-depth comparison between the performance of native Hadoop and 100% virtualised Hadoop instances including HDFS storage on the server DAS.  The table below is a comparison of jobs run on the physical nodes and again with a 1:1, 2:1 and 4:1 VM to physical server ratio. 

What is apparent from the graph above is that in the various scenarios tested the virtualised instances aren’t too far away from the native performance (depicted by the 1.0 on the Y axis).  In fact in the Terasort test (sorting data generated and writing out the results) the 4:1 configuration actually performed better,  which is quite interesting but may be down to test harness subtleties.

There is only so much you can read into these kind of whitepapers,  in my view it’s a yes, you should definitely consider virtualisation of Hadoop.  On the flip side, it’s essential that you do your own testing to ensure that your virtualised Hadoop solution delivers against your requirements whatever they may be.

What about Vendor support?

Support for virtualised Hadoop is another pushback I’ve heard on a few occasions and I have to say I understand that position.  Before my time at EMC I was a Solutions Architect implementing solutions in a UK based investment fund manager.  One of my team’s main mantra’s when designing solutions was to always ensure supportability,  so when I hear a customer complain about lack of support for a solution I often have to sympathise with the position they find themselves in.  They want to do something,  but can’t because it compromises the supportability of the business application.

Now this lack of support is a situation I’ve seen before,  it always used to be virtualising Oracle that used to get the push back.  Oracle wouldn’t support the virtualisation of their database as consolidation ultimately eroded their license revenues (per CPU licensing).  I personally think we’re simply seeing a similar thing occurring now with Hadoop and customers are therefore wary of building a solution that the Hadoop vendors won’t support.  I get that,  doesn’t mean it’s acceptable though!

What can I do about it though?

We at EMC had seen a bit of this sort of thing recently, specifically with Isilon for HDFS.  Hadoop vendors weren’t keen on losing revenue (as per my second example earlier) and weren’t really willing to support Isilon or sanction it’s usage, but customers were keen to use Isilon instead of the traditional DAS approach.  Now EMC obviously has alliances with all the major Hadoop vendors, that sadly doesn’t necessarily constitute support.

Now some of you may have picked up on the recent joint announcement between Cloudera and EMC around supporting Cloudera Enterprise Hadoop and Isilon for HDFS.  This is a great step forward in my opinion and has primarily come about following concerted pressure from Cloudera customers in the global financial industry.  Their sheer desire to leverage an enterprise grade platform for HDFS in tandem with Cloudera’s Enterprise Hadoop capabilities resulted in Cloudera having to agree to work jointly with EMC to build a supported solution.

Never underestimate your power as a customer,  most of the great things we come up with at EMC come about from our interactions with customers and them telling us what they need or want.  You shouldn’t be afraid to ask your vendors for the support you need for your business.

Cloudera Enterprise Hadoop and EMC Isilon for HDFS

Very briefly, the main plus point from a Cloudera / EMC point of view is around the fact that Isilon supports multi-protocol access of the same data,  eradicating the need to do major Extract, Transfer and Load (ETL) activity.  The same data can be interacted with via SMB, NFS, HTTP, SWIFT, REST and HDFS 1.0 and 2.2, ( 2.2, 2.3 and 2.4 coming very shortly) which allows you to put data into Isilon via one method, e.g. logs from an application and then consume it through another, e.g. HDFS.   My personal plus point is the fact that you can use Isilon HDFS for Cloudera, Pivotal HD, HortonWorks or Apache Hadop as your not tied to a traditional DAS stack

Some of the related Links to that announcement are included below,  with the Cloudera CTO blog post a particularly good read.

Cloudera Enterprise Hadoop and EMC Isilon supported solution
Cloudera’s CTO Blog Post on joint EMC Isilon solution
Cloudera Enterprise Hadoop and EMC Isilon solution overview

In Summary

So for those customers out there who want to virtualise Hadoop but want to ensure they are fully supported, I implore you to put the pressure on your Hadoop vendor and your VMware account team.  I know for a fact that VMware have alliance teams working with the Hadoop vendors,  but it needs real customer pressure on the Hadoop vendors to fundamentally change the game.  It’s what happened with Oracle on VMware,  it’s what happened with Cloudera and Isilon HDFS and it needs to happen for Hadoop on VMware as well.

It’s also worth noting that this isn’t just a Hadoop problem,  we’re going to end up in the same situation with loads of the new popular commercial open source variants .  Think of things like MongoDB (NoSQL) or DataStax (Cassandra DB), it’s only a matter of time before large enterprise customers are going to want to virtualise or use enterprise storage platforms with these technologies.

We at EMC aim to offer choice to customers,  so they are free to mix and match our technologies with whatever systems they want to put on top of them.  However I think we as a mainstream vendor need to do more work partnering and certifying our products to work with the ISV ecosystem.  I’m not saying we do this for all ISV’s,  we’ll need to be selective but I think we’ll reap the benefit of having done the work with Cloudera and Isilon.  We should ensure that it doesn’t stop there,  we should listen to our customers and we should aim to provide supported ISV and EMC stacks where needed.

EMC, Hadoop, Storage, VMware Cloudera, EMC, Hadoop, HDFS, Isilon, VMware

So,  I have to start off by thanking Brian Gracely (a fellow EMCer) for starting me off on this runaway train of thought.  a few weeks ago I read his article entitled “Will IaaS+ happen before PaaS” the question posed in the title is very apt, a question which only reinforces the fact that we work in such a rapidly changing IT world.  One where the battle for market and mind share continues to alter the landscape on a monthly if not daily basis.

IaaS, PaaS, SaaS…   Reality!!

I remember when I first started at EMC as a vSpecialist,  we were all about building private Infrastructure as a Service (IaaS) capabilities.  Virtualise everything on VMware, utilise vCloud Director to create pools of resource,  securely divide them between tenants, deliver a self service catalogue through a portal and conduct chargeback against all of it.  We also talked about extending that private cloud to the public cloud through a federated hybrid cloud model, but ultimately when you strip away the bells and whistles what we were all talking about from an end user perspective was the basic provisioning of VM’s to the OS layer.

Roll forward 3 years and things have changed up a gear significantly. Platform as a Service (PaaS) has become the new hot topic and the ability to write applications without worrying  about the underlying infrastructure is well and truly upon us.  Pivotal Cloud Foundry, Red Hat OpenShift, AWS Elastic BeanStalk, Microsoft Azure and SalesForce.com’s Heroku are some of the key players in today’s PaaS market. Some of these PaaS offerings are open and can sit on top of multiple cloud infrastructures,  others are more proprietary and locked in. All of them however are trying to capture a share of the customers who broadly speaking fall into the following categories.

Innovate or die — PaaS offers a way to “leapfrog” the competition with the ability to quickly integrate the latest innovations in software development and scale them quickly. Customers get that pie-in-the-sky seamless experience, which is a win for everybody.

Agility is key — PaaS is a strong entry point to embracing the DevOps mindset with minimal investment, helping organizations work toward agile development. When you don’t have to worry about the underlying infrastructure, it becomes a lot easier to achieve continuous deployment and quick, responsive feature updates. Developers don’t need to handle operations and operations don’t need to know how to code in order to take advantage of a PaaS.

Build once, deploy anywhere — This is relatively specific to the open source players, but the ability to build an application on a common platform means that you can write it once and deploy it on any infrastructure you’d like. In other words, if you build an application on Cloud Foundry, that application will run the exact same way on any other instance of Cloud Foundry, with the implication that you can ideally move from the public cloud to the private cloud or between public cloud providers with lessened fear of lock-in.

Quoting – Director of Pivotal’s Cloud Foundry Product Team Dekel Tankel in CiteWorld

The reality (in my opinion)… well it’s a mixed bag of course. Today I still see customers / IT teams striving to provide basic IaaS for their internal users.  In truth developers wanted that simple capability from internal IT over 2 years ago, it is one of the main reasons that many public cloud providers, such as AWS are what they are today.  They offered a viable and quick alternative for infrastructure while internal IT teams were either still trying to work out how to do IaaS themselves or were simply asleep at the wheel not realising that some very real competitor existed out there.

PaaS is an exciting shift in the industry, it’s allowing businesses to move up the stack, forget about the infrastructure and concentrate on building the applications that differentiate them from their competitors.  It’s still early days for PaaS,  though the momentum is building, not least with the Cloud Foundry foundation announcement this week which saw some of the  industry heavy hitters commit to developing the Cloud Foundry project.

I certainly don’t think anyone can argue with the concepts of PaaS, or the fact it will rapidly take share in  the coming years as development methods change.  I do however feel that it won’t suit everyone immediately,  It’s great for greenfield start-ups who are not constrained by legacy IT and want to operate in the public cloud, but how will it be adopted into the existing businesses? How quickly will it be adopted into the enterprise?  All enterprise customers should be taking a look at this today,  working out how they can integrate PaaS into their IT function to fundamentally change how they manage the software development lifecycle.

I personally think we’re at another one of those interesting inflexion points.  The business and the developers that work for them want to move ever faster.  Internal IT has maybe only just got to grips with IaaS and can now service them with the VM’s they want quickly.  However the developers have moved on and already want more,  they want to consume services and not just VM’s,  they look to the public cloud and they see rich services being layered on top of cloud infrastructure,  messaging services, Database as a service, in-memory capabilities, etc.  The developers and the business are again demanding more than internal IT are currently delivering,  sounds familiar doesn’t it!  Sounds like the IaaS story from a few years back all over again.

Delivering Iaas, IaaS+ and PaaS

The question I asked myself after reading Brian’s article was what are EMC doing to assist our customers to deliver in this crazy fast moving IT world of ours.  When I say “customers” I mean that in the broadest sense,  it could be assisting small IT departments,  enterprise customers or service providers looking to deliver services back to businesses and public consumers.

At EMC we’ve been talking a lot about the 1st, 2nd and 3rd platform. Some of you may have seen the picture below before,  sums up today’s modern IT world very well I think.  I certainly speak to customers who operate somewhere in all three of these platforms and I can safely say that I don’t see that model changing overnight,  however I do see companies striving hard to leave the legacy behind and leapfrog straight into the 3rd platform.

If we break it down, what businesses are technically looking to achieve today is optimising the 2nd platform and at the same time enabling the 3rd platform.  This is where I think a blended model of IaaS, IaaS+ and PaaS will cover the majority of use cases.  IaaS and IaaS+ will help optimise legacy and new 2nd platform applications and change how they are delivered.  IaaS+ and PaaS will find itself used for new application requirements in both the the 2nd platform and 3rd platform.

A picture speaks a thousand words,  so I’ve attempted to draw out below what’s in my head on this subject. Thanks to Eric Wright for the graphic inspiration in his recent blog post. In theory by mixing traditional IaaS with PaaS and looking to utilise that combined stack to layer services on top of IaaS,  whether that be PaaS delivered (MemDB aaS or Messaging aaS) or more traditionally delivered app blueprints (DBaaS straight on top of IaaS) we eventually come up with a hybrid model that caters to lots of different requirements.

So if we then take that on one step, what does the EMC federation have to offer in this space to help customers achieve this blended model of IaaS, IaaS+ and PaaS?  I came up with the below diagram, pretty busy isn’t it!!

Lets break it out from the top down

Services – there is a plethora of services that theoretically could be consumed,  some you may put together yourself (via application blue prints or custom buildpacks for Pivotal CF) others may be pre-packaged for you (either blueprints or packaged services).  They may be consumed on PaaS (buildpacks) or may be deployed straight onto IaaS (blueprints).  Pivotal services such as GemFire, RabbitMQ and TC Server are some of the offerings that can be deployed on either today.

vCloud Automation Center – vCAC has recently had the vFabric Application Director functionality folded in can be utilised to deploy VM’s, Operation Systems and application blueprints straight onto physical infrastructure, multiple hypervisors and multiple cloud platforms.  I’ve included Puppetlabs on the diagram as the integration with vCAC has greatly expanded the capability for service deployment, with vCAC being able to take advantage of the puppet modules library.   I think once VMware get vCAC fully plugged into Pivotal CF and the vCloud Hybrid Service (hopefully both aren’t too far away) it will make it an exceptionally powerful tool for automation whether that be VMware or a heterogeneous cloud environment.

Pivotal Cloud Foundry – The open source Platform as a Service offering that is today compatible with multiple cloud environments (VMware, OpenStack, AWS and I’m hoping vCloud Hybrid Service soon).  It currently comes in an Open Source and a Pivotal enterprise flavour, other custom variations will undoubtedly appear, IBM Bluemix is a recent PaaS based on CF that I’ve been reading about.  Cloud Foundry is creating a real stir in the IT world at the moment with a lot of the IT heavyweights throwing their backing behind this open source project, prompting comments such as the one below.

“Cloud Foundry is on an absolute tear. The number of companies that have bought into the initiative, the amount of code being contributed, the customer wins that ecosystem members are enjoying suggest that Cloud Foundry is preeminent among all the open source PaaS initiatives.”

— Ben Kepes, Forbes

VMware Software Defined Data Centre – I think everyone knows the story with this, if you don’t you really should   VMware in combination with it’s partners (EMC included) are working hard on delivering the software defined data centre.  The basic software defined compute layer is where VMware earned it’s stripes, that area of the SDDC needs no introduction.  Software Defined Networking became a mainstream topic with the VMworld 2013 announcement of VMware NSX.  With NSX VMware are bring the same consolidation, control and flexibility benefits to the network world as they did to the server world. Granular network policies that can follow a VM around regardless of it’s location (Private or Public clouds) is a key factor in enabling wide spread SDDC and hybrid cloud adoption. The last element is Software Defined Storage, something VMware and EMC are working very hard on.  EMC announced our ViPR offering at EMC world in 2013,  a fundamental change in thinking around storage.  Abstracting the control plane and enabling a single point of management for EMC storage, other vendors storage, as well as commodity and cloud storage was a major change of direction for EMC.  Providing software only data services is another fundamental shift in mindset, but an essential one as the storage world slowly becomes more commoditised.  Today EMC offer HDFS and Object data services through ViPR,  in the future there will be a lot more as EMC focus efforts on producing more abstracted software features.  VMware have also gone down the commodity route with VSAN,  still in Beta but due for release soon,  it will prove popular for those VMware only shops who want to consume commodity server direct attached storage.

VCE vBlock – The leading converged infrastructure stack company created as a joint venture between EMC, Cisco, VMware and Intel.  Converged infrastructure stacks are the easiest means of quickly deploying private cloud infrastructure within your own data centre. Built from best of breed components, available in multiple T-shirt size offerings, fully integrated, built in the factory, certified and ready to roll in 45 days or less,  it is the perfect way to consume infrastructure and underpin a blended IaaS, IaaS+ and PaaS model.

vCloud Hybrid Cloud (vCHS) – This week saw VMware announce their new vCHS service in Europe,  based in the UK.  This is a key announcement for VMware and one that will sure to get customers excited.  Lots of customers have VMware in their own private cloud deployments today,  enabling those customers to extend their internal VMware environments while use the same automation and monitoring tooling will be very appealing.  The capability to move your VM’s back and forth between on-prem VMware private cloud and vCHS or simply deploying your existing VM templates directly into a public cloud offering without upheaval is a huge plus or vCHS.  With IPSEC VPN or direct connect network capabilities on offer,  vCHS offers the simplest means of extending of your data centre to consume as required.  Once the offering beds in and a few more bells and whistles are added (I’m thinking Pivotal CF PaaS here) it will be an even more compelling offering and a large part of VMware’s future business.

Summary

This has been a very long post, written over a number of weeks and written about a moving target (I’m thinking Cloud foundry foundation announcement and vCHS launch here).  It basically comes down to businesses and the developers wanting to constantly innovate and do more.  It’s about IT struggling along to keep up with that insatiable appetite and deliver what they want.  I believe IT departments wants to help them do that, but they have to balance delivering with all the challenges that comes with existing IT Legacy and the requirement to maintain secure and compliant environments as per the rules and regulation that governs their business.

I believe the answer is a mixture of platform 2 and platform 3 solutions,  it’s a mixture of legacy and new world applications, it’s a mix of legacy IT infrastructure, IaaS, IaaS+ and PaaS as I outlined above. With the work going on at EMC + VMware + Pivotal it’s a no brainer that this particular federation of companies is in a perfect position to help the businesses, developers and IT Infrastructure teams with that journey to innovate and change how they do what they do!

EMC, Gestalt-IT, Pivotal, VMware EMC, IaaS, PaaS, Pivotal, SDDC, SDN, SDS, vCHS, ViPR, VMware, vSAN

The storage industry is undergoing a huge change at the moment,  a huge change that is moving at break neck speed and one that I find both exciting and scary all at the same time.  Exciting because it’s new technology being used to solve complex business and IT problems and scary because it’s moving so incredibly fast. I think you can only guess what’s going to happen next, which direction it’ll go in and which trends will win out at the end of the day.  I guess that’s probably why I’m writing a blog post for the first time in almost 2 years!  Yes I know, that is far too long between blog posts and yes I am extremely ashamed of myself!

I’m lucky enough to work for EMC and as part of that I get to see a lot of this change in IT developing momentum from the inside.  That said, in an organisation as big as this I also sometimes don’t find out about some of it until it hits the headlines, either at one of our product mega launches, a big industry event or in one of Chad’s now infamous blog posts!

Today’s Modern Storage Architectures

New storage architectures are appearing thick and fast,  my boss (Chad Sakac) explains this exceptionally well in his recent VMworld 2013 blog post on all things SDS at VMware and EMC.  He calls out 3 separate storage architectures that are fast becoming the norm,  they can be seen below and can be explained from left to right simply as.

Type 1 = Traditional storage model.

Type 2 = Scale out  / node based build out storage architectures.

Type 3 = Next Gen scale our software defined storage with COTS hardware.

EMC is very well known for the Type 1 storage,  VNX is a great example of this model in action.  EMC is also heavily into the Type 2 storage,  whether that be scale out NAS in Isilon, scale out block storage using the scale out engines of our enterprise VMAX arrays or even scale out block storage from an All Flash Array such as XtremIO.

It is however the type 3 storage I have been viewing with most interest and it is impossible to ignore the rapid increase in the use of the term “Software Defined Storage”.  I should really correct myself here though,  it’s not being used simply as a term or IT buzzword,  there are actually some very tangible products coming to market and this area of storage will gather huge momentum as we enter 2014. 

A subject that plays in this field and comes up regularly with one of my larger enterprise customers is Cloud Storage and Open Source Software with Commodity Off the Shelf hardware (COTS). It’s an interesting discussion and one that has been born out of both advances in Cloud storage platforms and  open source software stacks (OpenStack, Ceph) and a desire to introduce cost effective, supported  “good enough” solutions to support business demand. 

This stuff is real,  some people (including EMC colleagues) dismiss it as a passing fad,  something that will never take off as it’s not supported,  won’t work,  companies can’t put data in the cloud / on that platform for compliance or security reasons.  It’s seriously time to wake up and smell the coffee,  there may be challenges be they business or technical ones,  but the desire is there and where there is a will, there is always a way!

So where is this blog post going exactly… 

"Good Enough" In-Memory storage alternative with software defined storage

…So all of this leads me to an interesting use case that I was discussing both internally and with a customer recently,  one that covers off three very interesting areas. 

1. New Memory Channel Interface Flash (think NAND Flash with a DDR3 Memory interface)
2. Software Defined Storage pooling of that MCI based storage using EMC ScaleIO
3. A “good enough” performance tier, less than in-memory but more than PCI-E flash cards.

It sits firmly in the type 3 storage architecture discussed earlier, it involves new hardware technology combined with Software Defined Storage and could be used to deliver a cost effective, supported, good enough storage offering to deliver an alternative to in-memory databases.  Sound interesting? let’s dig a bit deeper into the 3 areas.

1. Memory Channel Interface (MCI) Storage

MCI storage popped up on my radar about a month ago following some internal distribution list emails.  A little digging shows that this particular technology appears to be in the very early stages of adoption.  In fact some of the tech companies in this field are still hiring ASIC programmers,  so it’s definitely early days.

One of the main companies out there at the moment is Diablo Technologies who are In essence providing non-volatile flash attached to a DDR3 memory interface to deliver up to 400GB of storage with a response time of less than 5μs.  The following is an extract from the Diablo press release which says it all really.

Configuring Memory Channel Storage (MCS) as a traditional block storage device enables new performance levels for applications, all the while reducing latencies by more than 85% over PCI-Express based SSDs and 96% over SATA/SAS based SSDs. MCS latencies are not only significantly lower than any other flash based storage, but also provides deterministic latencies with minimal variability, allowing applications to commit data quickly and consistently.

As well as being used for storage they the MCS DIMMs can also be used as an expansion of system memory,  allowing TB of addressable system memory instead of GB.  That’s another use case for another day though,  right now I’m interested in this as super low latency block storage and it sounds like this could be the next big thing in server based flash solutions.

2. Software Defined Storage Pooling with EMC Scale I/O

At present the problem I see with the above technology is the fact that because it’s basically direct attached storage (DAS) there is no protection of the data.  So you’re looking at stateless deployments or utilising an overlay to provide some host level clustering of the DAS resources.  This is where ScaleIO comes in…

ScaleIO in it’s simplest terms is a software defined storage solution that allows customers to leverage the local disk in applications servers in a pooled manner.  Through the installation of the ScaleIO agent known as the ScaleIO Data Sever (SDS), local storage can be combined together to create large scale elastic pools of storage.  Some of the key benefits are listed below

ScaleIO technically converges the storage and application layers and can be used in a number of different configurations.  Hosts are of course interconnected,  this can be achieved using Ethernet or Infiniband (In my example I’m thinking high performance back end here). As ScaleIO deployment is small the SDS (100Mb – 400Mb) and the SDC (20Mb) can be deployed side by side and all the assets in a server can be sweated to deliver a complete solution.

You can however configure things in a traditional 2-tier setup as shown below,  where applications utilising the ScaleIO Data Client (SDC) access storage from dedicated storage nodes using the Scale IO Data Server (SDS) agent.

 
In reality,  you can configure your application and ScaleIO in any way you like,  depends on your requirements.  Add into this mix the flexibility of agnostic support for physical and virtual nodes, full protection zones,  quality of service, 2 and 3 way data protection and the upcoming storage pools for performance.  It’s quite a neat little product.

So let’s put this back into the context of the use case I was originally discussing.  By taking a number of hosts containing the MCI low latency block storage,  connecting those hosts with Infiniband and adding ScaleIO over the top I’ve got myself a pretty interesting low latency,  highly parallelised IO storage system using commodity off the shelf hardware.  What’s not to like?

3. Cost Effective / Good Enough in-memory alternative using MCI and Scale I/O

So the 3rd and final point,  the reason  that I was discussing this topic with a customer in the first place was cost efficiency.  They were interested in how you could do provide a good enough in-memory alternative,  both in terms of software and hardware.  The following is an interesting extract from a blog,  in this instance they’re looking at using the MCI as a memory extension and layering on software to utilise that memory. In my case I’m looking at it as a storage extension.  Regardless of the use case though, the point is the potential for cost saving and there is one if you put some thought into it vs the traditional models.

http://www.gridgain.com/blog/why-mci-means-rapid-in-memory-computing-adoption/

Another important characteristic of MCI storage is the plug-n-play fashion in which it can be used – no custom hardware, no custom software required. Imagine, for example, an array of 100 micro-servers (ARM-based servers in micro form factor), each with 256GB of MCI-based system memory, drawing less than 10 watts of power, costing less than $1000 each.

You now have a cluster with 25TB in-memory storage, 200 cores of processing power, running standard Linux, drawing around 1000 watts for about the same cost as a fully loaded Tesla Model S.

Summary

This has been a bit of a ramble but I think what I am trying to say is,  the world is changing and there are now so many different ways to achieve your objectives.  The commoditisation of hardware and the major up shift in software defined everything is fundamentally changing the approach in IT.  As I said at the start,  it’s both scary and exciting all at the same time,  the only option is to embrace it if you ask me!

EMC, New Products, Storage MCI, Memory Channel Interface, ScaleIO, SDDC, SDS, Software Defined Storage

Since last week I have seen a number of questions in and around VASA and how it is configured for EMC arrays.  I got a couple while doing the Q&A for EMC’s recent VNX best practices with vSphere 5 live WebEx and the day after I was asked by Cormac Hogan over at VMware to take a look at a question asked on the VMware blog site.  So I’ll admit now,  I hadn’t really had a chance to look at VASA in-depth, shame on me!  However I thought that this was as good a chance as any to learn and I thought I would do a post on how to configure it for both EMC’s VNX and VMAX systems. Big thank you to my EMC colleague Garrett Hartney for providing both his time and an environment that we could set this up in.

EMC’s VASA implementation

For those not familiar with VASA I strongly suggest reading this article to familiarise yourself with the what and why around this new VMware API. For those who just want the short version, VASA is essentially an API that allows storage vendors to publish array capabilities up to vCenter.  This allows VMware admins to see characteristic information about the storage underpinning their datastores and also allows them to use VMware storage profiles to enforce VM storage placement policy compliance, e.g. This SQL VM will always sit on performance disks.

The table below shows how EMC currently publishes it’s array capabilities through VASA 1.0 up to vCenter.

An example of how this looks for a datastore when pulled through to vCenter can be seen in the screenshot below.

Core Components and architecture

Regardless of which array you are connecting to, EMC’s implementation of VASA is done using Solution Enabler and something known as the SMI-S provider.  Together these two components act as a middle tier between vCenter and the different arrays being queried.  It’s worth pointing out that the SE \ SMI-S server supports in-band (SAN attached) and out of band (network) connectivity for VNX and CLARiiON arrays and in-band connectivity only for Symmetrix arrays. The architecture of the setup is demonstrated in the diagram below.

SE / SMI-S server deployment

To get started with VASA you will need to download SMI-S version 4.3.1 which already comes pre-bundled with Solution Enabler 7.3.1.  This software can be downloaded from the link below and comes with an option for 32 and 64 bit Windows as well as Linux.  For full details on OS support see the release notes for SMI-S 4.3.1 – (PowerLink Account Required for downloads)

Home > Support > Software Downloads and Licensing > Downloads S > SMI-S Provider

As part of my own deployment I am using a Windows 2008 R2 64 bit server to deploy the core components.  The server has been built as standard with no special configuration required.

• We first of all need to deploy Solution Enabler and SMI-S on your designated server, locate the installation media and run the install package.

• When presented with the welcome screen click next.

• Leave the install location as default and click next.

• When prompted select the array provider only and click next.

• Review the installation settings and space requirements and click next to install.

• Once the install is complete, click finish.

• Configure the environment variables on the server to include the SYMCLI path

• Locate the following file and open it for editing

• Locate the line below and change the value from 100 to 1200, save and exit the file.

• Navigate to the services console and restart the ECOM service

• navigate to the location shown below and run testsmiprovider.exe

• the next step is to connect to the SMI-S provider. I used the defaults which are shown in the square brackets, just hit enter on each line to use the default.

• Once connected you will see the following at the command prompt

• type the dv command at the prompt to display version information about the SMI-S provider install.  This basically proves that everything is working as expected

• that concludes the basic installation and configuration of the SMI-S and Solution Enabler server,  now all we need to do is add in the storage arrays we want displayed to vCenter via the VASA api.

CLARiiON and VNX

SUPPORTED – CLARiiON Navisphere Release 28, 29 & 30, VNX Unisphere Release 31
(SMI-S supports many earlier CLARiiON releases but vSphere 5 does not)

Earlier I mentioned that the CLARiiON and VNX arrays could be added to SMI-S in-band or out of band.  The most common method and the one I intend to use here is to connect out of band, i.e. across the network.  If you do want to connect in-band with direct SAN connection then check out page 39 of the SMI-S v4.3.1 release notes.

One major pre-requisite for connecting CLARiiON and VNX is that the user account used to connect to the arrays must be an administrator login with global scope. At this point you should hopefully still be connected to the testsmiprovider.exe application used earlier,  if you are not then please repeat the command line steps shown above to reconnect.

• Once connected successfully type the commands addsys to begin adding the array
• Enter the IP address / DNS name for Storage Processor A and hit enter.
• Enter the IP address / DNS name for Storage Processor B and hit enter.
• You can continue to add additional arrays here or hit enter to move to the next step.
• Accept the default for the address type, i.e. IP/Nodename by hitting enter.
• Continue answering this question for each storage processor / array added
• enter the global scope administration user account for connecting to the arrays.
• enter the password for the administration account being used
• You will then see the message +++++ EMCAddSystem ++++

• After a while you will see the output from the addsys operation, as you can see below the output is 0 which indicates success.  The details of the system added are then listed.

• If you now run the dv command the arrays added will be listed as connected.

• Now that the array is registered we now need to add the VASA provider into vCenter. Log into vCenter and navigate to the home screen, locate and click on the storage providers icon.
  

• Within the storage provider screen click on the add button as shown below.

• Enter a name for the provider and enter the URL shown below,  the IP address of the server hosting SE / SMI-S should be entered where it has been blanked out below.  The user name is admin and the password is #1Password.

• When prompted accept the certificate for the SMI-S provider

• Once successfully added you will see the provider displayed

• Highlight the provider and you will see the array that was connected to the SMI-S provider server earlier.

• To check that VASA is working correctly in vCenter click the VM Storage Profiles icon on the home screen within vCenter.

• When setting up a new Storage Profile you should be able to see the storage capabilities presented to vCenter,  these are shown below and are marked with system.

• Job done, VASA successfully deployed and storage capabilities showing in vCenter!

DMX4, VMAX and VMAXe

SUPPORTED – Enginuity 5875
(SMI-S supports earlier Enginuity releases but vSphere 5 does not)

Now unfortunately I do not have access to a Symmetrix to complete my testing, however the release notes for SMI-S state the following which makes it sound very easy.

When started, the SMI-S Provider automatically discovers all Symmetrix storage arrays connected to the host on which the Array provider is running. No other action is required, such as running a symcfg discover command.

As mentioned earlier Symmetrix discovery is done in-band through small gatekeeper LUNs presented to the SE / SMI-S server.  If it is a virtual server then ensure that the LUNs are presented to the VM as physical mode RDMs.  The SMI-S release notes has the following to say about best practice.

When using the SMI-S Provider to manage Symmetrix arrays, it is recommended that you configure six gatekeepers for each Symmetrix array accessed by the provider. Only set up these gatekeepers for the host on which the SMI-S Provider is running.

So in theory it should be as simple as completing the following steps

• Present the gatekeeper LUNs to the server (physical or virtual)
• Restart the ECOM windows service to restart the SMI-S provider (auto discover arrays)
• Use testsmiprovider.exe tool,  run the dv command, verify Symmetrix array is attached.
• Thanks to my colleague Cody Hosterman (who does have a Symm) for the screenshot.

One point to note, if you have SMI-S installed on the same host as the EMC Control Center (ECC) Symmetrix Agent or Symmetrix Management Console (SMC) there are a couple of steps you need to take to avoid some spurious errors.  Check out page 37 of the SMI-S v4.3.1 release notes for further information on the changes required to avoid this.

Summary

I think the important thing to remember here is that this is version 1.0 of VASA. It may not be the most elegant solution in the world but it is a start on what I think will become a key feature in years to come. We are fast moving into an age where VMs become objects where we simply check a box to ensure our requirement or service level is delivered.  Imagine a scenario where a VM is created and as part of the creation process you select the storage based on the VASA information passed up to vCenter from the array.  Do I want it on a RAID 5 or RAID 6 protected datastore? Do I want it on a RecoverPoint replicated datastore? Do I want it on a vPlex distributed datastore? Do I want it on a datastore that is SRM protected?  Although it is v1.0 you can see the potential use cases for this feature in the future are going to continue to expand.

Some of you may well have seen Chad Sakac’s blog post back in September entitled Help us make VASA (and EMC’s VASA provider) better! It includes a questionnaire with questions about what you, the end customer wants to see from VASA. This is a great chance to have your say and influence how EMC implement VASA going forward, lets make v2.0 of VASA a feature that delivers on the huge potential V1.0 has shown.

EMC, VMware EMC, VASA, VMware, vSphere 5

I come into contact with a lot of IT products throughout my day job,  some are introduced to me by customers, some by colleagues and some by EMC Partners.  Monday was no different as I got chatting to an EMC partner who was sitting opposite me in the office, naturally the subject turned to the product his company makes.  The company in question is Axxana and the product they make is called Phoenix System RP™, a product that is designed to deliver zero data loss but in a very different way to the traditional Recovery Point Objective (RPO) = Zero infrastructure you’d expect.

Zero Data Loss = Synchronous Replication

Traditionally Zero data loss is delivered using synchronous replication technology and due to the costs involved it tends to be reserved for the most key mission critical systems.  With synchronous replication when an application writes data to the storage, that data has to be written to both storage locations before the application receives a write acknowledgement (see below).  As you can imagine when doing this between two physical sites application latency becomes a key consideration and as such these setups are usually backed by expensive low latency inter site fibre connections, not cheap!

It’s also worth noting that this latency consideration usually restricts the distance between the production and secondary site.  This can often still leave you exposed to possible outages, i.e. natural disasters that could impact both sites.  To mitigate this companies often replicate to a third site asynchronously, more leased lines, more storage, more management overhead and generally more expense!

That extra expense however can often be justified! In my previous role in the finance sector, trading systems or back end pricing warehouses were usually set up in this manner due to the potential cost of service loss or data corruption.  Data consistency and RPO was always the key requirement when recovering from an outage, RTO being the obvious runner up.  When talking to application owners the message I was often given was “as long as the data is correct I don’t care how long it takes you to get it back”.  Obviously they did care about RTO, but recovering a system in 1 hour only to find that the data is inconsistent was not an acceptable outcome post outage.

Zero Data Loss = Asynchronous Replication

So how is Axxanna different?  what is it they do that allows for zero data loss while using asynchronous replication. Well first of all it’s important to point out that this product integrates with EMC’s RecoverPoint replication product to provide the asynchronous replication.  RecoverPoint is a product that works by splitting the write I/O for any protected LUNs, journaling it, compressing and de-duping it before replicating it with write order fidelity on the target storage location.

Axxana adds another level to this process which you can see in the diagram below. The first step is to mark a RecoverPoint consistency group as Axxana protected. Once this has been selected the writes that are usually just synchronously written to the local RecoverPoint appliance (RPA) are also then written synchronously to the Black Box via the Axxana collector servers. The collector takes the block stream adds some consistency checking meta data and then encrypts and writes the data out to the Black Box for safe keeping. An acknowledgement is only sent back to the application once the write has been committed to both the RPA and the Black Box in order to guarantee zero data loss.  At the same time the RecoverPoint appliance is replicating the data asynchronously across to the DR site as normal.  The key point here is that a combination of the Asynchronous replicated data to the second site and the data held within the Axxana Black Box on the primary site can be merged to create the equivalent of a synchronous data set at DR.

The Phoenix Black box can contain up to a 300GB SSD so it is capable of storing a lot of RecoverPoint data.  This capacity makes it a perfect solution for protecting against WAN failure scenarios as well as data centre / application / storage failure scenarios.  While the WAN link is down the RecoverPoint data is being synchronously played into the Black Box thus maintaining your zero data loss DR Protection.

The disk capacity raised an interesting point for me,  how does the Axxana solution know to expire files from the disk inside the Black Box?  I dug a little deeper and spoke to someone at Axxana and they told me the following.

In the initial configuration of an Axxana protected CG, Axxana gets an initial lag size from RP and configures an initial buffer of the same size (+ 10%) on the Black Box SSD. The blocks received from the RP are written cyclically to this buffer. This way we maintain only the last blocks (the delta) in any given moment. The Black Box buffer size is adjusted dynamically according to the changes of the RP lag.

So it decides the space allocation based on the RecoverPoint lag, i.e. the amount of data waiting to be replicated to the secondary site.  Dynamically expanding that space allocation allows it to deal effectively with replication lag spikes or WAN link loss, pretty impressive stuff.

So that’s how it functions at a high level for the protection, the next question is what happens if my production site is hit by a disaster?

Axxana Black Box Construction

So I’m guessing that you’re now thinking how on earth am I going to recover the data if its stored on a piece of infrastructure in the Data Centre that has just been burnt down / hit by a plane / insert disaster here / flooded ?? Surely putting it in the primary data centre goes against all data protection logic! Technically you would be right, but you need to understand how this thing is constructed to see why that isn’t going to be a problem.

Axxana describe the Phoenix system as an Enterprise Data Recorder (EDR) based on technology from the aviation industry, i.e. plane black box flight recorders. It’s built as a hardened disaster proof storage device to ensure that the synchronous data held within it remains intact no matter what disaster befalls your data centre.

It’s constructed of 3 main layers, protection levels and pictures of the layers can be seen below.

Electronic Box – water protection
Cylinder – shock protection
Fire protection box – Well the name says it all really!

So while the rest of the data centre is a smouldering wreck you can quite happily set about retrieving your data for recovery at the DR site.

Data Recovery

So how do you get the data back in the event of a disaster, well that’s where things get  interesting and as someone who loves technology I think this next part is pretty cool.

First of all you need to physically locate the system, you do this by tracking the homing signal installed within the Black Box.  Once you have found it you can then connect a laptop with an Axxana software component installed and extract the data.   Now a physical connection is all well and good but what if the police or fire brigade won’t let you anywhere near the site let alone dig through the rubble looking for your Black Box!  Well that is where the 3G – 3.5G phone transmitters comes in handy,  allowing you to transfer the data from the Black Box using mobile phone technology.

The Black Box obtains an IP address from the nearest mobile phone base station and use it to communicate over the internet with the Axxana Recovery servers. The Recoverers can be either wired or wireless connected to the internet. Every interaction between the Black Box and the Recoverer is mutually authenticated using RSA 1024 bit protocol. all data that is sent over is encrypted using AES 128 bit protocol with a dynamic key exchange mechanism that automatically changes for every block of 32MB.

It’s all very clever stuff,  I have to admit I am impressed with both the concept and the end product itself, I would love to speak to someone that has used it in anger.

Summary

So this is a product I saw a few years back when I was a customer,  it was shown to me by EMC as part of a RecoverPoint sales pitch. I remember at the time thinking it was a pretty cool idea and a pretty full on cast iron way to guarantee the protection of critical data, however I couldn’t see a use case for it outside large enterprises. After talking about it for a while the other day I realised that back then I was potentially missing one of the key selling points.

You utilise Axxana so you don’t have to do expensive synchronous replication, so you don’t have to introduce unnecessary application latency, so you don’t have to have that second site within ~100KM distances.  The reason this product is built to withstand every feasible disaster is so that you can safely use cheaper asynchronous replication over large distance and still guarantee that synchronous replication RPO that the business or application owner demands.

I swear one of those imaginary light bulbs went on above my head while I was discussing it!

If you want to know more about this product check out the Axxana website or please speak to your EMC account manager about the product.  Alternatively you can drop me an email and I’ll find someone to talk to you about it.

EMC Axxana, EMC, RecoverPoint, Replication

I’ve recently been looking at the implementation of EMC’s free Virtual Storage Integrator (VSI) with a few our older Symmetrix customers. Now customers using VMAX and VMAXe have the ability to deploy delegated storage provisioning for their VMware admins. However DMX customers only have the ability to use the read only storage viewer functionality as the DMX is not supported with Storage Pool Manager (SPM) which back ends the storage provisioning.  Some interesting questions came up recently with a customer about how best to deploy the VSI storage viewer with DMX arrays and I thought it would be worth sharing the findings with a wider audience. Basically I’m looking to cover off the different ways the VSI can connect to a Symmetrix array and how some of the options selected affect end to end operations.

VSI to Symmetrix Connectivity

So the VSI tool can be used in two ways with Symmetrix arrays,  you can utilise the local Solution Enabler (SE) installation that comes with the VSI or you can use a dedicated Solution Enabler server. It’s important to remember that Symmetrix arrays can only be discovered in-band, basically this means the SE install needs direct connection with the physical array.  This is achieved through the presentation of small LUNs known as gatekeeper LUNs, something existing Symmetrix storage teams will be very familiar with. So lets look at the two different possible setups.

Local Solution Enabler Deployment

The local deployment model shown above would require a gatekeeper LUN being presented / zoned through to the machine that the VI Client, VSI and local SE install have been deployed on. Communication with the array in this instance flows directly between the client PC and the array. In the majority of instances this isn’t going to be very practical for a number of reasons.

• Each VMware admin client with VSI deployed would need a direct array connection.
• Most Symmetrix arrays are FC attached and client PC’s are not.
• Arrays live in remote data centres and VMware admin PC’s live in the office.
• Increased security risk, i.e. too many direct array connections to secure

Remote Solution Enabler Deployment
 

The remote deployment model shown above would require gatekeeper LUNs being zoned through to a dedicated server. VMware admins would then connect through this remote SE server when querying Symmetrix arrays from information with the VSI. Communication flow in this instance always goes through the server, however as you’ll see later results can be returned from the SE server or the array depending on VSI configuration. This kind of setup is more practical for a number of reasons.

• Remote SE servers are usually already in place for storage management tasks.
• Available as a virtual appliance for quick deployment if not in place already.
• Supports connectivity by multiple remote VMware admins using VSI.
• Manage multiple Symmetrix arrays through one server.
• Decreases security risk, i.e. single device connection to array.

Mix and Match

The model above is by no means rigid,  you can craft a number of solutions out of the principals shown above.  If your vCenter server sat in the same data Centre as the array then you could present gatekeeper LUNs to it and use this as a management point whenever you want to get information from the array.  Another possible solution is to put a management virtual machine in the datacentre with the VI Client and VSI installed and present a gatekeeper as an RDM,  whenever a VMware admin needs information from the array they connect into that management VM to carry out the work. Basically there is a solution for deploying VSI with Symmetrix arrays no matter what you’re setup looks like.

VSI Discovery Process Flow

One question that did come up recently was what happens when you select the AutoSync option for a symmetrix array and you are using the remote SE server solution. How often does it poll the array?  Well the answer is it doesn’t, which is strange as the term Autosync gives the impression that it syncs with the array on a regular basis. So how does it work?

AutoSync enabled

When AutoSync is enabled each time you request array data,  e.g. clicking on the EMC VSI tab for a datastore.  The request forces the SYMAPI database on the remote SE server to be updated from the array, the up to date array information is then returned to the VSI. There is obviously a slight cost involved in doing this as the remote SE server needs to make the calls to the array in order to update it’s local database before responding.  Typically this would introduce a 10 – 20 second delay but that cost means you guarantee the information received is up to date and valid.

AutoSync disabled

When Autosync is disabled each time you request array data the request is returned from the cached information in the local SYMAPI database on the remote SE server. This is obviously the fastest method as you don’t have the cost of querying the array directly for an update but the information may be out of date.

With Autosync disabled it’s up to the VMware administrator to initiate the sync of the array from within the VSI.  Alternatively the storage team can initiate sync with the array directly through the SE server using SYMCLI. To initiate a sync manually go into the VSI tool and select Symmetrix arrays from the list of features, highlight the array and click on Sync Array.

Summary

The free EMC VSI Storage Viewer tool can be of great benefit to Symmetrix customers, allowing VMware admins improved visibility of the underlying storage layers.  In larger environments where Symmetrix arrays are traditionally used you tend to find VMware and Storage are managed by separate teams.  Anything that improves the information flow between the two teams during troubleshooting has to be a must have tool. As show above some thought needs to be given to how you set it up. My personal preference would be to always go for the remote SE server solution.  Enable Autosync if your underlying VMware storage environment changes often and if it doesn’t then a manual sync every now and again should suffice.

Additional notes and links

SPC-2 Flags

It’s worth noting that SPC-2 flags need to be set on the FA port or on the initiator of the ESX host for the VSI to work correctly, in fact this is a required setting for ESX generally. This has come up a couple of times recently so I though it worth mentioning to ensure that people have it setup correctly,  the following whitepaper gives you more information.

http://powerlink.emc.com/km/live1/en_US/Offering_Technical/White_Paper/H4116-enabling-spc2-compl-emc-symmetrix-dmx-vmware-envnmt-wp.pdf

VSI installation Media

Home > Support > Software Downloads and Licensing > Downloads T-Z > Virtual Storage Integrator (VSI) – (Please Note: PowerLink account required)

Solution Enabler Media

Home > Support > Software Downloads and Licensing > Downloads S > Solutions Enabler(Please Note: PowerLink account required)

Solution Enabler Documentation

Home > Support > Technical Documentation and Advisories > Software > S > Documentation > Solutions Enabler – (Please Note: PowerLink account required)

Storage, VMware EMC, VSI

At EMC the vSpecialist team often end up talking to a lot of customers about EMC’s FREE Virtual Storage Integrator (VSI) Plug-ins for vCenter Server.  Not only do customers love the fact that it is FREE they also love the features delivered. The ability to accurately view, provision and manipulate EMC storage directly within vCenter empowers VI admins and makes everyone’s life that little bit easier.

When I started writing this article we were on version 4.2 of the VSI plug-ins, following VMworld 2011 we are now up to version 5.0 the fifth generation of this excellent VMware / EMC toolkit. The plug-ins that make up the VSI are listed below, to download use the link below or use the cookie trail to navigate to the page on EMC PowerLink.

• VSI Storage Viewer Plug-in 5.0
• VSI unified Storage Management Plug-in 5.0
• VSI Storage Pool Management Plug-in 5.0
• VSI Path Management Plug-in 5.0

Home > Support > Software Downloads and Licensing > Downloads T-Z > Virtual Storage Integrator (VSI) – (Please Note: PowerLink account required)

One of the great features that people are drawn to is the ability to allow VI admins to provision storage directly from within vCenter. This is done with the VSI Unified Plug-in for Celerra, CLARiiON and VNX(e) and done with the VSI Storage Pool Management plug-in for the VMAX. One of the first question I often get asked is how is the secured,  how does the storage team ensure that only the right VMware admins are manipulating the underlying storage?

The answer previously was… well to be honest we didn’t really have an answer to this one. Technically if you allowed the VMware admins to provision storage you needed to trust them not to go provisioning crazy and fill up your storage array.  Obviously that response was not really acceptable for any environment and EMC have been working to rectify that.

The Access Control Utility is a new part of the VSI framework which allows storage administrators to granularly control availability of storage platforms and storage pools on those platforms.  These security profiles when created can be exported and passed to the VMware administrators and imported into the VSI unified storage management plug-in. The following blog post details the steps involved in completing this process for a VNX array in vSphere 4.1

So we start by double clicking on the shiny padlock icon that will have been added to your desktop when you installed the VSI unified storage management plug-in.  When the ACU starts we are presented with the profile management screen.  This will of course be blank the first time you start the utility, in this screenshot below however you can see a couple of existing access profiles I have created for some VNX arrays in the lab.

To Create a new profile you simply click the Add button, you are then presented with the details screen for the new access profile being created.  Here you enter the name of the profile and a suitable description and click next when finished.

The next step in the wizard is where you define the storage system that will be permissioned as part of the security profile.  You click on Add and then select the system you are going to permission,  as you can see the VSI ACU supports Celerra, CLARiiON, VNX and the VNXe arrays. For VMAX you need to look at Storage Pool Manager (SPM) to control access,  I’ll look to blog about this one at a later date.

The next screen presented very much depends on the storage system you select.  If you chose the Celerra option you’re prompted for the details of the control station, username and password.  Select the CLARiiON and you’re prompted for the Storage Processor details and login credentials. If you select the VNXe then you’re promoted for the management IP and the login credentials.  I’m sure you can see the pattern developing here!

In this example we are dealing with a VNX array and as such the option is whether you want to give access to block storage, file storage or both. As both are controlled differently within the VNX, if you select both you will need to enter the IP and credentials for the Storage Processor (Block) and the VNX Control Station.  For the purposes of this example I’m going to use Block only as you can see in the screenshot below.

When you click next you’re prompted to enter the storage processor IP address and log on details as shown below.

Once you are authenticated you get to select the granularity of access you want to provide.  It’s important to note that when the ACU refers to storage pools it means any storage pools and traditional RAID groups that may have been created on the VNX array.  There are 3 options available as you can see in the screenshot below.

• All storage pools
  This option basically gives a VMware Admin free reign to provision LUNs with the VSI all over the array.  A potential use case for this may be a dedicated development VMware environment with its own dedicated array where the storage team don’t care to much about usage.

• No Storage Pools
  This option is a complete lockdown and acts as an explicit deny to prevent any accidental provisioning on an array, i.e. the VSI unified storage management feature cannot talk to the array full stop, it won’t even show up as an option.

• Selected storage pools
  As the name indicates this option allows the selection of certain storage pools for VSI provisioning.  A potential use case here would be a mixed environment where the array is shared between VMware and physical workloads.  As a storage administrator you would grant permission to the VMware storage pools only thus preventing any potential mis-provisioning (not sure that is actually a word but it certainly has its place when we talk about VSI provisioning)

In this example I’ve chosen selected storage pools as I think this is probably the scenario that most people will be looking for the ACU to help them with.  Within the next screen you are presented with a list of all storage pools / RAID groups on the array.  Here you select the storage pools / RAID groups you want to give the VMware admin access to, when your happy with your selection you simply select finish.  Note in the screenshot below that I have select two individual storage pools (one is a RAID group) to be part of this particular storage profile.

Once you’ve completed storage pool selection you are returned to the profile screen,  you can finish your profile creation right here by clicking on finish or you can add additional storage systems if your VMware environment consists of multiple arrays.

Once you have completed the creation of your security profile the next step is to export it so you can pass it over to your VMware admins. To do this simply highlight the Security profile, click on export and save the file

Chose a location to save the file and don’t forget to add a passphrase to the file so that it cannot be misused.

It’s important to remember that the login credentials provided by the storage admin during the ACU profile setup are the ones used when the profile is imported into the VSI.  The VMware admin will see the connection details and username being used but will not see the password. For audit purposes on the array it may be best to setup a dedicated account for use with the VSI and storage profiles. It should also be noted that the full details of the storage profile are encrypted within the profile export file as you can see below.

So now that you’ve finished creating your storage profile you can pass it on to the VMware administrators to import into the VSI.  To do this you go into vCenter and open up the EMC VSI screen from the home screen.  Click on the Unified Storage Management feature,  then click on add and select Import Access Profile before clicking next.

You now select the XML file created by exporting the ACU storage profile, you enter the passphrase you selected and click next.

As you can see below the VNX array has been added to the VSI and provisioning access is marked as Restricted.  This is as expected as we configured the profile to give access to only two storage pools, FAST_Pool_3_Tier and RAID Group 10.

When you use the EMC VSI to provision storage you will be presented with the VNX array that was part of the imported profile.  You select the storage array and as you can see in the screenshot below you can only create storage on the two storage pools that were added to the ACU storage profile.

Summary

The EMC Access Control Utility was something I have been looking to write about for a while. Since it’s release I’ve often wondered how exactly it worked, what it could / could not do and how it could better meet customer needs. The steps above show that it is possible for a storage team to delegate control of storage pools so VMware admins can quickly provision the storage that they need. Becoming more efficient is something we as vSpecialists talk about on a daily basis, this tool is one of those first steps that you can take to make life easier.  If you are a VMware admin who is working with EMC storage then I suggest you speak to your storage team about this.  Likewise if you are a storage admin, reach out to your VMware counterparts and discuss how this could save you both time in the long term.

Video

My boss Chad Sakacc put a video together for VMworld 2011 which maybe explains it better (certainly quicker) than I maybe have in this blog post. I left it to the end though so you read the article before discovering it . My step by step approach is simply so I can fully understand how it fits together and as I go deal with the many “what if” or “how does that work” kind of questions.  Hope you find it useful in some way, feel free to comment or ask questions.

EMC, New Products, VMware ACU, EMC, VMware, VSI

Since I started at EMC just over 2 months ago I’ve been spending a lot of time getting to grips with the large range of products that EMC has in it’s portfolio.  One key product I’ve been lucky to spend some time learning about it is VMAX / Symmetrix.  A Product range I knew a little about but had never used as a customer or had the chance to deep dive technically. Luckily for me I got the chance to do the deep dive with 3 days of VMAX training with some of the Symm engineering team from the US.

During this VMAX training some of my more “Symmetrix savvy” colleagues (David Robertson and Cody Hosterman) were telling us about something called EMC StorReclaim. At the time I couldn’t say anything about it as it wasn’t due for unveiling until EMC world but I did take notes with the aim of following up after EMC world.  I only found those notes today hence the delay folks!

First things first,  this is an EMC internal tool for EMC Global Services usage. I am publishing this in order to bring it to your attention.  If you feel you have a need for using this EMC product then please speak to your EMC Rep / TC for more details.

So what is EMC StorReclaim? I could explain it myself but this extract from the release document explains it perfectly.

StorReclaim is a Windows command line utility designed to free allocated but unused storage space as part of EMC’s Virtual Provisioning solution.

StorReclaim determines exactly where the allocated but unused space is and then passing that information on to Symmetrix for space reclamation. Once the storage capacity is reclaimed, it is then put back into the device storage pool for consumption. The process is performed in real time and does not require any running application to shutdown.

So what does it support / not support and what can I run it on?

Host operating system requirements
StorReclaim is fully supported on the following operating systems with various Service Packs.

◆ Windows Server 2003 x86/x64
◆ Windows Server 2008 x86/x64
◆ Windows Server 2008 R2
◆ Windows Server 2008 R2 with SP1

StorReclaim also supports Windows Guest OS running on:

◆ Hyper-V Server 2008 or 2008 R2
◆ VMware ESX Server 3.5 or 4.0
◆ VMware vSphere client 4.1.1

Note: For VMware ESX, the physical disks in a virtual environment must be attached to virtual machines using RDM (Raw Device Mapping). For Microsoft Hyper-V, the physical disks must be configured as pass-throughdisks.

File system requirement
Microsoft Windows NTFS
MBR & GPT
Basic Disks
Dynamic Concatenated, Mirrored (excludes striped / RAID 5 Dynamic disks)

Logical volume managers requirement
Microsoft Windows LDM

Storage environment support
StorReclaim supports Symmetrix arrays running Enginuity 5875 and higher.

Supported with EMC Clone and SNAP but only de-allocates storage on the source

Just to clarify one of the points around virtualisation support. This tool does support both physical and virtual windows server workloads.  Key point here is that the virtual machine in question must have RDM attached disks served from a thin pool.

One other key point worth mentioning is that this tool does not require you to install Solution Enabler or any other EMC host based software.  The tool works via a windows filter driver and sends SCSI UNMAP commands directly to the VMAX array in order to return the blocks to the thin pool.

As I mentioned earlier if you want access to this tool you will need to obtain it from EMC Global Services.  I hope that this will be released as a customer tool at some point in the future, this decision may well be based on demand so please ask if it is something you could use.

EMC, New Products EMC, Storage, Symmetrix, VMAX

A common theme that I see coming up up time and time again with customers is VDI using Citrix XenDesktop and vSphere 4.1.  It’s popularity generally stems from the previous success companies have had with the more traditional Citrix products such as Presentation Server / XenApp.  I know when I was looking at VDI solutions I was very much in favour of Citrix due to one thing, the ICA protocol.  It works and it works well over long distances, in a lot of companies it has proven itself over a long period of time, it is a protocol they trust to deliver.

Following a customer meeting recently I was desperately searching for an EMC reference architecture (RA) for a XenDesktop / vSphere deployment.  At the time it turned out we didn’t have a completed one,  we did however have one in draft format that was going through the final stages of review.  That RA has now been completed and released for public consumption, an overview of the documents purpose is below.

The purpose of this reference architecture is to build and demonstrate the functionality, performance and scalability of virtual desktops enabled by the EMC VNX series, VMware vSphere 4.1 and Citrix XenDesktop 5.  This solution is built on Machine Creation Services (MCS) in XenDesktop 5and a VNX5300 platform with multiprotocol support, which enabled FC block-based storage for the VMware vStorage Virtual Machine File System (VMFS) and CIFS-based storage for user data.

The RA covers the technologies listed below and details why the VNX array with FAST cache enabled is a perfect match for your Citrix VDI deployment.  One other interesting area that is discussed is the use of Citrix Machine Creation Services (MCS) which is a new feature XenDesktop 5 and provides an alternative to Citrix Provisioning Server (PVS).  For those new to MCS I suggest you have a read through the following Citrix blog post as their are some additional design considerations around IOPS that need to be considered.

• Citrix XenDesktop 5
• Microsoft Windows 7 enterprise (32-bit)
• Citrix Machine Creation Services (MCS)
• VMware vSphere 4.1
• EMC VNX 5300 – (Fast Cache & VAAI enabled)
• EMC virtual storage integrator (VSI) – Free on EMC PowerLink

If you are considering XenDesktop 5 and vSphere 4.1 then I suggest you download and have a read through the RA linked below.

EMC Infrastructure for Virtual Desktops enabled by EMC VNX Series (FC), VMware vSphere 4.1 and Citrix XenDesktop 5

Citrix, EMC, vSphere Citrix, VDI, VSphere