Virtualization vendors may still be touting the potential of putting 20, 50 or even 100 virtual machines (VM) on a single physical machine. But IT managers and industry experts say those ratios are dangerous in production environments and can cause performance problems or, worse, outages.
"In test and development environments, companies could put upwards of 50 virtual machines on a single physical host. But when it comes to mission-critical and resource-intensive applications, that number tends to plummet to less than 15," says Andi Mann, vice president of research at Enterprise Management Associates Inc. in Boulder, Colo.
In a 2009 study of 153 organizations with more than 500 end users, EMA found that, on average, enterprises were achieving 6:1 consolidation rates for applications such as ERP, CRM, e-mail and databases.
Thevariance between the reality and the expectations, whether it's due to vendor hype or internal ROI issues, could spell trouble for IT teams. That's because the consolidation rate affects just about every aspect of a virtualization project -- budget, capacity and executive buy-in. "If you go into these virtualization projects with a false expectation, you're going to get in trouble," Mann says.
Indeed, overestimating physical-to-virtual ratios can result in the need for more server hardware, rack space, cooling capacity and power consumption -- all of which cost money. Worse yet, users could be affected by poorly performing applications. "If a company thinks they're only going to need 10 servers at the end of a virtualization project and they actually need 15, it could have a significant impact on the overall cost of the consolidation and put them in the hole financially. Not a good thing, especially in this economy," says Charles King, president and principal analyst at consultancy Pund-IT Inc. in Hayward, Calif.
Why is there a disconnect between virtualization expectations and reality? King says that up to this point, many companies have focused on virtualizing low-end, low-use, low-I/O applications such as test, development, log, file and print servers. "When it comes to edge-of-network, non-mission-critical applications that don't require high availability, you can stack dozens on a single machine," he says.
Bob Gill, an analyst at TheInfoPro Inc., agrees. "Early on, people were virtualizing systems that had a less-than-5% utilization rate. These were the applications that, if they went down for an hour, no one got upset," he says.
That's not the case when applying virtualization to mission-critical, resource-intensive applications -- and virtualization vendors, on the whole, have been slow to explain this reality to customers, according to some analysts.
Once you consider applications with higher utilization rates, greater security risks, and increased performance and availability demands, consolidation ratios drop off considerably. "These applications will compete for bandwidth, memory, CPU and storage," King says. Even on machines with two quad-core processors, highly transactional applications that have been virtualized will experience network bottlenecks and performance hits as they vie for the same server's pool of resources.
Here are four tips for avoiding server overload.
Start With Capacity Analysis
1. Start With Capacity Analysis
To combat the problem, IT teams have to rejigger their thinking and dial back everyone's expectations. The best place to start: a capacity analysis, says Kris Jmaeff, information security systems specialist at the Interior Health Authority, a British Columbia government agency.
Four years ago, the data center at Interior Health was growing at a rapid clip. There was a lot of pressure to virtualize the 500-server production environment to support a host of services, including DNS, Active Directory, Web servers, FTP, and many production application and database servers.
Before starting down that path, Jmaeff first used VMware tools to conduct an in-depth capacity analysis that monitored server hardware utilization. (Similar tools are also available from Cirba, Hewlett-Packard, Microsoft, PlateSpin and Vizioncore, among others.) Rather than looking at his hardware environment piece by piece, he instead considered everything as a pool of resources. "Capacity planning should focus on the resources that a server can contribute to the virtual pool," Jmaeff says.
Already, the team has been able to consolidate 250 servers -- 50% of the server farm -- onto 12 physical hosts. And while Jmaeff's overall average data center ratio is 20:1, hosts that hold more-demanding applications either require much lower ratios or require that he balance out resource-intensive applications.
Jmaeff uses a combination of VMware vCenter and IBM Director to monitor each VM for telltale signs of ratio imbalances such as spikes in RAM and CPU usage, or performance degradation. "We've definitely had to bump applications around and adjust our conversion rates according to server resource demand to create a more balanced workload," he says. If necessary, it's easy to clone servers and quickly spread the application load, he adds.
"Because we did our homework with ratios of virtual servers by examining the load on CPU and memory and evaluated physical server workloads, we've been pleasantly surprised with our ratios," Jmaeff says.
2. Monitor Performance Continuously
At Network Data Center Host Inc., a Web service provider in San Clemente, Calif., the IT team quickly learned that when it comes to virtualizing mission-critical applications, you have to consider more than just RAM. "We originally thought, based on available RAM, we could have 40 small customers share a physical server. But we found that with heavier-used applications, it's not the RAM, it's the I/O," says Chief Technology Officer Shaun Retain.
The 40:1 ratio had to be pulled back to no greater than 20:1, he says. To help with that effort, the team has developed a control panel that allows customers to log in and see how their virtual machines are handling reads, writes, disk space usage and other performance-affecting activity. In addition, NDC Host uses homegrown monitoring tools to ensure that ratios aren't blown by a spike in a single VM's traffic.
3. Test for Application Stability
King says companies should also conduct rigorous testing on their virtualized mission-critical applications before and after deployment. "You have to make sure that in terms of memory and network bandwidth, each application is stable at all times. For instance, if you know an application is harder-hit during certain times of the year, you'll want to account for that in establishing your ratios," he says.
Testing will also help IT teams determine which virtual workloads will coexist best on a physical host. "You have to make sure that a physical server isn't running multiple VMs with the same workload. Otherwise, if they're all Web servers, they will be contending for the same resources at the same time, and that will hinder your consolidation ratio," says Nelson Ruest, co-author of Virtualization: A Beginner's Guide and founder of Resolutions Enterprises, a consultancy in Victoria, British Columbia. Instead, IT staffers should make sure that workloads are heterogeneous and well balanced based on peak usage times and resource demands.
Ruest also warns IT teams not to forget the spare resources that host servers need so they can not only support their own VMs, but also accept the workload from a failing host. "If you're running all your servers at 80%, you won't be able to support that necessary redundancy," he says.
Most organizations will find that they need to dedicate at least a month to the capacity planning and testing phases to determine the appropriate physical-to-virtual server ratio for their environment, Ruest says.
4. Get Real-world Benchmarks From Peers
Finally, EMA's Mann advises IT teams to get involved in local user groups or attend large annual conferences, like VMware's VMworld or Citrix's Synergy, to meet peers with similar application environments. "Most attendees are more than willing to share information about their environment and experiences," he says. Rather than relying on vendor benchmarks, get real-world examples of what has worked and what hasn't at organizations with profiles similar to yours. "You'll have a better chance at setting realistic expectations."
