Adventures In Managing Virtualization

Evan Jafa, chief information officer of First American, says getting your virtualized servers set up and running right really is just the start of any IT leader's virtualization work. And that if you don't think holistically about virtualization, you're in for a rude surprise. "I think lots of folks are going to find out that virtualization poses a whole new set of concerns for security, networks and applications," says Jafa. In fact, he calls management a make-or-break issue for virtualization. "Monitoring and management becomes absolutely critical in the virtual environment," says Jafa, who started work about three years ago on a large project to consolidate First American's data centers and standardize its storage and networking technologies.

To that end, his organization is directing much of its energy today at management processes and tools, wide application of ITIL process to the virtualized environment, and even the re-shaping of IT staff. Jafa hopes that in the future virtualization will play into better management of his networking technologies as well. He's already testing some new technologies to advance that goal. If you're trying to do holistic virtualization planning, Jafa's advice may prove particularly interesting.

Big Provisioning and Disaster Recovery Wins

First American's business, which posted $8.2 billion (about Rs 32,800 crore) in  2007 revenues, delivers information such as mortgage, real estate and financial data. So while many IT leaders closely monitor virtualization projects to ensure that  application service levels don't slow down for internal users, Jafa's team also keenly focuses on application performance for external customers. The company has two datacenters with about 4,500 physical servers, soon to be 6,500 (due to new business units). It serves 40,000 to 50,000 concurrent users during peak business hours; its network carries 10 to 21 terabytes of data per hour, averaging about 15 terabytes per hour, Jafa says.

To keep all that data flowing smoothly, Jafa says his tool chest currently includes offerings from VMware, Microsoft, BEA (WebLogic) and "lots of Linux". His team is also using HP's SiteScope tools for hardware performance monitoring. Jafa says he defines virtualization as "achieving the highest efficiency of any given system at any given time". In servers, virtualization is netting him 50 to 60 percent utilization rates, compared to 10 to 15 percent on standard servers, he says. The team also sees a 30 to 35 percent reduction in energy consumption per virtual machine compared to its physical counterpart, Jafa notes.

To date, IT has virtualized 25 percent of their servers, Jafa says. These include production servers and test and development and staging boxes. In some cases, entire business units now have had all their servers in a virtual environment, he says. "My goal is to reach 50 percent virtualized this year." "Any server that IT's using is going virtual," Jafa says, which means some 300 servers will be consolidated onto seven or eight physical machines soon. "We continue to work on virtualizing disaster recovery." Jafa says he is also bringing DR in house: "That's going to be completely virtualized", he says, noting he hopes to complete the disaster recovery project by the end of the year. Virtualization has helped Jafa's team as the company centralizes IT services for an array of BUs.

Politics and Process Lessons

"The process of bringing them on and migrating them has become faster," Jafa says. The company's IT provisioning has improved by a factor of 65 percent. On the DR and high availability front, virtualization has given Jafa equally dramatic results. For every 100 servers, he now puts aside four servers for failover and six to eight servers for workload help at peak usage time. Previously, the company's high availability arrangement meant that he needed to put aside 200 servers for the same purpose.

Politics and Process Lessons

But server virtualization is only the start of the changes - not the end. "As we started to think about virtualization, we started to think about standardization," says Jafa, as in standardizing processes and utilizing shared resources, across not only servers but also storage and networking technology. In the course of doing his planning, Jafa says he has run across two key challenges that other IT leaders have cited in a recent CIO survey, Your Virtualized State in 2008: workload balancing and IT staff management issues. "Capacity planning has become critical in the virtual world," Jafa says. "That one we're still maturing and figuring out." He says he hasn't had to deal with much squabbling among IT members who are being asked to work together in new ways. The bigger political challenge, he says, involved First American's diverse collection of business groups.

"We have many business units and to get them off the notion of 'my servers' and 'my storage' took a lot of work," Jafa says. "I credit our (IT) change management organization." Still, he says, he faces a wave of organizational questions, ones that he feels most IT leaders will have to battle soon. "Virtualization will pose a serious challenge organizationally to IT," he says. "The current vertical disciplines that we have will not be appropriate." For example, he says, IT may need a horizontal group with members from various teams to handle provisioning. How is his group dealing with this situation in his own organization? "So far we haven't torn down the vertical organizations," Jafa says. "Through the architecture organization, we have brought those teams together."

His IT team now has an overall architecture team that works on design, architecture and process questions, he says. Process, he believes is key. Jafa, who was already a big fan of using ITIL to standardize IT process before he started the virtualization efforts, weaves ITIL into his team's efforts in the physical and virtual worlds. Jafa says close to 55 percent of his team is ITIL-certified and he has a full-time ITIL instructor in house . "In the virtual world, processes become super critical," Jafa says. "We want to make sure to test our processes to see how they will do. For example, provisioning has to be constantly defined," he says. "We just did a whole assessment of exactly what will the impact of virtualization be on our processes," he says. "We have to meld some of our processes for the virtual world," he says, noting network design and even software design has to change in the virtualized environment.

"Now we can look at the long term impact." On that networking front, Jafa is now working with Cisco to test out technologies such as Virtual Switching Systems (VSS) and VFrame (a family of provisioning and automation tools), in a quest to make his network more standardized and simpler to manage - essential for the long term. "Network provisioning becomes as critical as server provisioning," Jafa says. "We are very interested in virtualization of the LAN, so we don't have all these protocols for storage and fibre in the network. The simpler the protocols and the more collapsed the protocols are, the easier the virtualized world will be managed."