Choices Matter – The Final Chapter

Stephen Manley

Stephen Manley

CTO, Data Protection and Availability Division
Over the past 15 years at both EMC and NetApp, I have traveled the world, helping solve backup and recovery challenges - one customer at a time (clearly, I need to optimize my travel arrangements!). My professional mission is to transform data protection so that it accelerates customers’ businesses. I have a passion for helping engineers pursue technical career path(without becoming managers), telling stories about life on the road and NDMP (yes, that’s NDMP).

No series of blog posts, regardless of wittiness, insight and general brilliance, can provide a prescriptive answer about how to protect your VMs. (Of course, if you do find some blog posts with wittiness, insight, and brilliance, please send them; I’d love to see how it’s done. Except for Chad’s posts. Don’t send those.)

The three customer stories in this series demonstrate the disparity I see in our customer base. People are choosing different approaches based on their organization’s current challenges – technical, business or organizational. While everybody’s road will be different, I do believe the ultimate destination will be a versioned replication solution. Thus, as you plan your approach, I encourage you to explore the trends driving backup and recovery to chart your own course:

  • Deduplicated disk remains the first step. It enables you to optimize traditional backups, adopt tools like Avamar, build brute-force versioned replication and evolve to versioned replication solutions, without changing your storage infrastructure.
  • Backup and recovery windows will continue to put pressure on data protection solutions. The best way to scale will be versioned replication. You need to decide the layer on which to build the data movement intelligence: backup client, storage, application, or hypervisor. Each layer offers unique characteristics in granularity of management (per-VM, per-app or per-storage container?), flexibility (do I want to lock in on one storage array for both primary and protection storage?) and functionality (how important are traditional backup features?). If you have not considered versioned replication, now is the time. If you have already drawn some conclusions, the optimizations in the VMware and application segment make this an ideal time to explore this area again.
  • More administrators – not just backup – will play an active role in protection. Users demand better service and more visibility into their protection schemes, and the only way to meet expectations will be to embrace technical changes like versioned replication. Also, as one company discovered, technical change will depend on organizational change. Backup teams will need to partner with other IT teams (e.g. VM, application, or storage teams) or these teams will find other ways to meet their protection needs. Monolithic backup solutions and teams will not succeed.

Change isn’t easy, but it is necessary. When you combine your insight of new technology trends with your users’ critical needs – you can lead your organization and deliver value. Or, you can be trampled by the wave of new technology. Server virtualization is leading a new wave of technology for backup and recovery. The choices you make today will define your future. They matter.

As for my daughter (for those of you who have been following this series since the start), she eventually put on a pink dress with black polka dots and reveled in the adoration of the masses: “Oh aren’t you just the cutest thing ever?” Clearly, the right choice.

In the Words of Customers – Customer Story #3 – Hybrid of Versioned Replication and Traditional Backup

Stephen Manley

Stephen Manley

CTO, Data Protection and Availability Division
Over the past 15 years at both EMC and NetApp, I have traveled the world, helping solve backup and recovery challenges - one customer at a time (clearly, I need to optimize my travel arrangements!). My professional mission is to transform data protection so that it accelerates customers’ businesses. I have a passion for helping engineers pursue technical career path(without becoming managers), telling stories about life on the road and NDMP (yes, that’s NDMP).

During World War I, Will Rogers, noted American humorist, was asked by a young reporter how he would stop the U-boat attacks on ships, to which Rogers responded, “Boil the ocean.” But when pressed for how he would do that, Rogers quickly replied, “I’m just the idea man here. Get someone else to work out the details.”

For the first five years of my career, I subscribed to the Will Rogers’ school of problem solving. If customers would wipe out their legacy infrastructure and immediately deploy our newest products across their environments, their problems would disappear. But that was before I met real customers with real data centers and, well, in the immortal words of Keanu Reeves, “Whoa!”

In this blog series, we’ve talked about the variety of mechanisms to protect VM environments and met two mid-sized customers that have deployed innovative solutions to solve their business problems. In this post, you’re going to meet a large customer that described its biggest challenge with one word – “change.” Its complex legacy infrastructure, organizational silos and users’ unwillingness to embrace new initiatives have made it difficult for the company to successfully change anything.

The company, which prides itself on minimizing exposure to cutting-edge technology (its IT team slowly rolls out new technology, allowing it to demonstrate stability and business value and gather user demand before rolling it out more broadly), has successfully deployed thousands of VMs. The company began by virtualizing its test/dev environment and lower-tier applications. But once manufacturing discovered that it could spin up a new application on a VM within 48 hours (versus a 90-day procurement cycle on physical servers), the IT team had its first advocate and the company’s path to virtualization was paved. The company is now virtualizing its business-critical applications.

Following the IT team’s standard technology deployment model, the backup team at this company has deployed new backup technology on an “as needed” basis. Backup and recovery performance has been the forcing function for change. Initially, it protected VMs just like it did physical servers – with a traditional backup to tape. However, when that approach failed to scale, the IT team evaluated two alternatives: guest-level VM backup via Avamar and traditional backups via VMware Consolidated Backup (VCB).

Ultimately, the company chose Avamar. While the initial Avamar roll-out addressed only the heavily loaded ESX server running the Tier-3 apps, the main driver for the remainder of the company was the network. The cost of local network bandwidth made it too expensive to push thousands of full backups across the LAN. Nearly two years later, it uses Avamar for 95% of its VM backups.

However, the company faces new backup and recovery window challenges. With even more heavily loaded ESX servers, Avamar guest-level backups struggle to complete. While it is backing up only the unique, new data, it’s expensive to search through the VMs to find that data. Therefore, as before, it has selected a new technology solution and is selectively deploying it.

The solution: Avamar-driven VMware Changed Block Tracking (CBT)-based versioned replication on the same heavily loaded ESX servers that first deployed the Avamar clients two years ago. While the company is concerned about the scalability of CBT and VM performance, change – specifically, organizational change – is the real issue. How can it trust a backup solution that depends on technology and administrators who are in different groups?

Fortunately, the company has a safety net. Avamar enables it to implement and manage both solutions – Avamar and CBT – from one interface. (“And if this VMware CBT thing fails, we can quietly switch back to the existing approach and not look stupid.”)

Rapid recovery, not rapid backup, has created a groundswell of demand in the organization. With Avamar’s Changed Block Recovery, on three separate occasions, the IT team has recovered applications within minutes. In each case, the data suffered a logical (read: user-driven) corruption. In the past, the company would either have had to run a full restore to an alternate VM or restore all of the files that comprised the application, but with VMware CBT-based versioned replicas, it simply rolled back the changes to the point of the last backup – moving a fraction of the data. (“It was like having a high-end snapshot storage system for all our apps – even the so-called tier-3 apps.”)

Currently, fewer than 10% of this customer’s VMs are protected by VMware CBT-based versioned replication. And while the backup team continues to roll out the new technology as deliberately as possible – incrementally building a relationship with the VM team – it acknowledges the power of VMware CBT-based versioned replication. The company has solved backup and recovery performance problems that seemed intractable before, and users (“for once”) are leading the drive for change in the company.

We all know that boiling the ocean does not solve problems in real customer environments. For most people, change doesn’t start with a clean sheet of paper, it starts with a mess. Therefore, like this customer, you need to set a clear goal and make a path that moves you in that direction. Along the way, you’ll need to embrace new technology, new techniques, and new organizational structures. You likely won’t overcome your VM backup challenges with one grand proclamation – you need to win hundreds of individual battles. In a more pragmatic moment, Will Rogers also said, “Even if you’re on the right track, you’ll get run over if you just sit there.”

Making the Right Choices – Part IV – In the Words of Customers

Stephen Manley

Stephen Manley

CTO, Data Protection and Availability Division
Over the past 15 years at both EMC and NetApp, I have traveled the world, helping solve backup and recovery challenges - one customer at a time (clearly, I need to optimize my travel arrangements!). My professional mission is to transform data protection so that it accelerates customers’ businesses. I have a passion for helping engineers pursue technical career path(without becoming managers), telling stories about life on the road and NDMP (yes, that’s NDMP).

Do you remember the anticipation of the conclusion of the serialized stories you loved best? Maybe you waited with breathless anticipation for Harry Potter to finally defeat Lord Voldemort, for Luke Skywalker to defeat the Empire or to find out whether Michael Knight and KITT could save the day in any of the Knight Rider cliffhangers. While you may not feel the same tingle about the conclusion of this blog series, it cannot be any less coherent than the end of the Matrix Trilogy.

In prior posts, I talked about the importance of making the right choices, the challenges of providing scalable and efficient protection of VMs as well as some of the technologies (e.g., VMware Changed Block Tracking, versioned replicas, etc.) that IT folks need to be thinking about.  But enough theory – what are companies really doing? Not surprisingly, the customers I’ve met have all chosen different approaches to protecting their VMs. However, despite the variations, there are two common themes: displacing tape with deduplicated disk and innovating to meet backup/recovery windows.

Over the next three posts, I will cover three different customer deployments.

Customer Story #1: From Storage-Based Replication to VMware CBT-Based Versioned Replication

Prior to joining EMC, I spent over a decade helping NetApp build its storage-based versioned replication suite (e.g., SnapMirror, SnapVault, Open Systems SnapVault, etc.). I worked with a variety of exceptional customers.  At a recent conference, one of my more excitable former customers saw me, and shouted, “I replaced all your old stuff (i.e., SnapVault) with all your new stuff (i.e., EMC backup and recovery technology). It’s so much better!”  To be honest, I wasn’t sure how to react. On the one hand, I was pleased my new company was able to improve his backup environment, but on the other hand, I had written much of the Snap code and none of the EMC code. (I quelled this threat to my ego by focusing on my wisdom in joining a group of people much smarter than me.)

Even without this riveting blog series, this customer had realized the potential of VMware CBT. Originally, he had provisioned VMs on FC-LUNs on FAS systems. He retained one week of nightly snapshots on the primary system and one month of nightly snapshots on a SnapVault secondary. He was thrilled with his backup performance and reliability, and had only three complaints about his environment:

  1. “Wasting” nearly 50% of space on the primary flexvol because he would not implement thin provisioning (fractional reservation < 100) and could not support automatically deleting snapshots (they were his backups, after all). He feared that he would run out of capacity and bring down his VMs. Without thin provisioning, his first snapshot reserved half the space on his flexvols to ensure that he could overwrite every block in the LUN without failure.
  2. Single VM recovery was complex and inefficient, since he could not put each VM in its own LUN, much less its own flexvol.
  3. Granular recovery from the VMs was complex and inefficient.

With the release of VMware’s Changed Block Tracking (CBT), his deployment strategy changed. He opted to use Avamar 6.0 to run raw VM backups with VMware CBT directly to Data Domain systems, where he retains each nightly backup for a month. His entire environment backs up within 15 minutes with stellar success rates. (He told me, “I ain’t gonna jinx myself by saying 100% success rates. You can’t make me do it. So stop trying. Just stop.” Of course, at this point, I hadn’t said a word in 10 minutes.)

He has successfully rolled back a VM to a prior night’s backup, using Avamar’s Changed Block Recovery, restored a VM to a new location and extracted a variety of spreadsheets and documents from the backups. While he has been thrilled with the performance, simplicity, and security of the Avamar solution, its cost-savings capability was clearly the highlight. “It’s saved me more money than anything else I’ve done all year,” he proclaimed.

He was able to:

  1. Regain 50% of his FAS capacity by embracing thin provisioning. He still uses snapshots, but since they’re not “official” backups, he’s enabled automatic snapshot deletion to free up space for the VMs, if necessary.
  2. Repurpose the SnapVault secondary FAS as a primary storage system.
  3. Reduce the size of the backups by 3x (the additional space savings provided by Data Domain’s compression and variable-length deduplication vs. NetApp’s snapshots and deduplication).
  4. Reduce network bandwidth consumption by 1.5x – less data means less bandwidth required.

Not only did the Data Domain systems pay for themselves, but the savings funded a separate project to protect his mainframe environment (and this isn’t the first time I’ve seen VM optimizations fund mainframes – who knew?). He now has backup storage for 3 years, primary storage for 2 years, and bandwidth for the next 12 months. This was not a three-year ROI – it was immediate.

In the next post, we’ll examine a customer implementing VMware-versioned replication in a way that blends brute force with subtle elegance (think either of the dancing hippos in Fantasia or Chris Farley as a Chippendale on SNL).

Next-Generation Storage and Backup for vSphere 5

Phil George

Phil George

Avamar/VMware Guru, Data Protection and Availability Division
Working with customers and partners (like VMware) to develop leading backup solutions makes every day very interesting; helping them optimize their backup architectures for virtualized environments is what really energizes me. Over the past 25 years, I’ve held senior engineering, marketing and sales roles within the technical software industry. This gives me a good vantage point to recognize technical challenges, see emerging trends and propose new solutions. I hold a BSEE from Cornell University and a Masters in Computer Engineering from Boston University. I currently reside with my wife and two children in Massachusetts.

By Phil George, Senior Product Marketing Manager,  Avamar

VMware vSphere 5 is a major release that brings a wide range of improvements and new capabilities – ideal for virtualizing mission-critical applications. These include up to 32 virtual CPUs with 1 TB of RAM, VM high availability and better security. As you undergo your IT transformation, it’s time to evaluate current storage platforms and backup and recovery processes. Be proactive and you will end up saving time and money.

Traditional storage and traditional backup are slow, complex and costly in virtualized environments. EMC provides optimized solutions for VMware that will enhance performance, simplify management and increase savings. Sound too good to be true? Please read on and you will get the facts – just the facts.

EMC’s next-generation unified storage, the VNX family, delivers 3x performance: The FAST Suite adds performance and maximizes storage efficiency by deploying this FLASH-first strategy. FAST Cache, an extendable cache of up to 2 TB, gives a real-time performance boost by ensuring the hottest data is served from the highest-performing Flash drives for as long as needed. FAST VP then complements FAST Cache by optimizing storage pools on a regular, scheduled basis. Together, they automatically optimize for the highest system performance and the lowest storage cost. Now that’s fast – and doesn’t that sound good? The VNX series platform is optimized for virtualization with over 75 points of tight integration to VMware. EMC Unisphere together with VMware vCenter Server and VMware vStorage APIs for Storage Awareness (VASA) make storage management in a virtualized environment a seamless experience.

EMC’s next-generation backup and recovery with Avamar deduplication backup software and system delivers 3x faster VMware backups and 30x faster restores. Avamar is tightly integrated to VMware vStorage API for Data Protection (VADP) and leverages VMware’s Changed Block Tracking (CBT) for faster backups and faster recovery. That’s right …CBT for faster recovery means faster time to productivity and we all know we don’t like waiting. Avamar is able to quickly determine the changed blocks from the last backup, then restores only those blocks to recover your VM. This means you get your lost, deleted or overwritten data back very quickly. Customers changing from traditional backup to next-generation backup reduce their total cost of ownership, streamline management, remove risk and accelerate their journey to the cloud.

Making the Right Backup Choices – Part III

Stephen Manley

Stephen Manley

CTO, Data Protection and Availability Division
Over the past 15 years at both EMC and NetApp, I have traveled the world, helping solve backup and recovery challenges - one customer at a time (clearly, I need to optimize my travel arrangements!). My professional mission is to transform data protection so that it accelerates customers’ businesses. I have a passion for helping engineers pursue technical career path(without becoming managers), telling stories about life on the road and NDMP (yes, that’s NDMP).

Previously on “The Backup Window,” I had described a scene that would rival a classic 1960s Batman cliffhanger. I was desperately trying to get my daughter dressed for church, so I could save my donut from the clutches of some Joker. Backup administrators tried to solve the Riddler’s question, “How can I protect all my VMs within my backup window?” As they looked at storage-based versioned replication, they worried that it would Freeze the flexibility of their virtualization environment. Fortunately, *Pow* *Zap* VMware Changed Block Tracking (CBT) was coming to the rescue.

However, despite its importance, CBT hasn’t generated the hype of Kim Kardashian’s wedding. (Google hits: VMware CBT – 2.2 million; Kim Kardashian Wedding – 18.5 million. I see you pretending not to know who Kim Kardashian is. Stop. You’re not fooling anybody. ) VMware CBT promotes versioned replication to become a first-class backup citizen. With VMware identifying the changed data, we can now build solutions that combine the best of storage-based-versioned replication with the best of the backup functionality.

  • Minimal load on the client. The server tracks the changed blocks through the day; the client then reads only the changed blocks at backup time.
  • Minimal load on the storage array. Reading only the changed data.
  • Light network load. Deduplicating changed data further reduces the network load.
  • Vendor lock-in. CBT works independent of storage array. Customers can back up VMs on any primary storage to any backup system that supports versioned replicas.
  • Full functionality. The backup application or VM-specific protection tool provides the full backup workflows and functionality you have come to expect.
  • Granularity of backup management. CBT works on a per-VM basis, so you can apply independent backup schedules, retention periods, security domains, etc. on each VM. More importantly, you can change those settings at any point, without penalty.

In short, you get the performance of versioned replication with the functionality, control and heterogeneity of backup.

What’s required to transform a technology like VMware CBT into a backup solution?
It’s bad enough trying to run recoveries from incremental file-based tape backups. Can you imagine the agony associated with trying to piece together something coherent from random series of blocks on tape? (Actually, I can – I watched the Tron sequel on cable this weekend. Yikes.)

Any viable CBT solution depends on a backup infrastructure built to manage changed-block versioned replicas, specifically:

  • Dedupe backup software. Rapidly transform, catalog and index changed data into full backups to optimize for recovery reliability, simplicity and performance. You do not want software with “bolted-on” virtual synthetics to become a bottleneck or point of failure.
  • Dedupe storage system. To store tens, hundreds or thousands of full backups, you need a storage system that can efficiently create and store those backups.  As with any backup system, make sure that you can trust it as the storage of last resort and that you can manage each backup version independently.

By combining VMware CBT with the appropriate backup infrastructure, customers have not only been able to reduce their backup pain but also increase the amount and type of production load on their ESX servers. Of course, I don’t want to be a shameless product shill, so I’ll let you figure out what types of backup solutions can properly leverage VMware CBT. (Editor’s note: EMC is not too shameless to point out that Avamar 6.0 uses VMware CBT to deliver this ideal versioned replication solution to either an Avamar Data Store or Data Domain target.)

While both I and the rest of The Backup Window have focused on the value of VMware CBT, wise readers observe that this is merely one instance of this technique. Oracle Block Change Tracking, Microsoft filter drivers and storage-based versioned replicas highlight a broader industry trend. Backup software needs help from the primary data owners to scale. You want to bet on solutions that demonstrate a commitment to integrating with all those data owners. (Because you’d like your backup solution to last longer than a Kardashian marriage.)

In my next post, I will share some customer deployments in which the use of backup solutions with VMware CBT helped remove roadblocks to VM deployments while bringing storage and backup teams together, ultimately showing why making the right backup infrastructure choices does make a difference. So, tune in to the next blog post – same blog site, shorter backup window!