FOSE Session: Implementing Virtualization - Enterprise Benefits and Challenges

April 2nd, 2008 by Louis DiMeglio

The second session handed out a book from its sponsor AMD: Virtualization for Dummies. There were noticeably less attendees in this session than in the Web 2.0 session. Are people more interested in Web 2.0 than virtualization?

Teresa Bozzelli from IDC Government Insights is the moderator, with Dr. Shawn Landry from reVision Inc. (stepping in for Hari Bezwada of the U.S. Army) and Bajinder Paul, Chief Information Officer from the Office of the Comptroller of Currency. Bajinder also worked at HUD as Deputy CIO, and Computer World recognized HUD for global best practices and serving constituents. Also a Fed100 winner.

Teresa: What are the differences between state and local agencies in regards to virtualization?

Shawn: There a lot of synergies between state and federal. Some of the challenges that separate them are the size of the federal level. There are a lot more intricate tendencies and difficulties associated with change. Scale drives complexity issues.

Teresa: Bajinder will start and speak about his experience at OCC. Shawn will follow up and focus on the Pentagon.

Virtualization in the Office of the Comptroller of Currency

Bajinder: You’re going to get the business aspect to why we are virtualizing our infrastructure. This is not about the technology itself, but about the business.

Based on the executive summary, the OCC is an independent agency of the Department of Treasury. We regulate and supervise national banks. 1,710 banks and about 50 federal branches of foreign banks. National bank assets are more than 7 trillion in assets, or 67% of the total U.S. commercial banking assets. The OCC also focuses on policy development and interpretation.

Virtualization is a key part in infrastructure optimization. The overall roadmap is needed to rapidly deploy information and services across the infrastructure at any time.

Virtualization. Why? Most of you are in IT, the bottom line is assets. We have applications, hardware, operating systems. As business needs increase, the assets you’re managing increase. You have to provide capacity on demand.

Why virtualize? Disaster recovery, low per server resource, etc.

What to virtualize? Production application servers, test/dev environments, file and print servers, underutilized web and application servers, firewall servers, and database servers with few to moderate users.

Over time because of emerging business needs.

In one weekend we have to backup 12 terabytes of data, bring down and re-start 240 servers. Huge implications for upgrades. We have to do this successfully and can’t have any downtime.

The question becomes, are there alternative strategies to manage these upgrades?

Virtualization at OCC improved responsiveness, quickly answered unplanned server requests, reduced acquisition timeframe, took advantage of “pooled’ capacities, and provided faster implementation.

It became a natural choice.

Development and testing cycles were improved with virtual machines and staging. It also improved communication options, with virtual desktop access and efficient utilization of limited bandwidth. From a development lifecycle and a user perspective, it’s all beneficial.

The bottom line, it doesn’t matter if you’re using AMD or HP or whatever technology, to have the right business - virtualization lets you quickly deploy applications.

One of the business outcomes was the ability to support community banks and make a complicated formula easier. They needed an application that allowed them to look at all the criteria for assessing the risk, which was the application that we deployed.

Our time from development to production was compressed by half. Less testing in different environments etc.

Part of the agency’s mission is to share information with FDIC and the Federal Reserve Bank. Again, having that virtualized environment and the ability to provision resources, helped that process.

We also do a lot of forecasting analysis. Examine risks and determine where banks are going. Business intelligence becomes extremely important.

As a CIO, virtualization is working because it’s reducing my lifecycle time. The bottom line, we are in a better position to manage and assess risks of the banking system.

Virtualization in the Pentagon.

Teresa: Over 21 billion was allocated in 2007 to IT management. How much of that is operations? IDC survey data says that over 65% of datacenter infrastructure cost goes to management and administration of people. 13% of that is power and cooling. Virtualization gives us a great opportunity to optimize the people side as well.

Shawn: When the Pentagon started a datacenter, it was back in 1998 to mandate that tenants at the Pentagon go to a common datacenter.

We realized we needed to provide more of a virtual datacenter.

The Pentagon renovation challenge:

• Space is a premium, power and cooling are a premium, resources at the Pentagon are a premium
• Server growth was overwhelming the construction schedule
• Mission support systems are limited (mission critical cooling and power)
• Duplication of many system components
• Inefficient utilization of datacenter space

So what does it take to build a virtualized datacenter?

The Pentagon chose to focus on changing the way people do business. Starting with:

• Access control and change management
• Remote administration
• Better metrics capabilities
• Complete capability of virtualization
  • Mainframe
  • Server
  • Fabric/Storage tape backup

The Pentagon also looked at how customers and tenants could focus on their mission and less on datacenter management. Tenants are still managing the virtual components, but no longer manage the storage systems, tape backup system, etc. They provide the requirements and the service agency provides the infrastructure to both manage and monitor the environment.

We were able to reduce power and energy costs by over 1,000 servers, and do 10 times the amount of computing. This has to do with airflow, cable management, and better manage our people. We had to put in tools to help solve problems and push people to get out of the datacenter, and comfortable with remotely logging in to the servers for administration. We met a lot of resistance to tele-commuting, but that has changed.

The Pentagon wants to virtualize everything, not just servers. Storage, content delivery, and overall enterprise architecture and business processes. They are headed towards implementing key processes and standards to virtualization, and making it work across the board.

Question and Answer Time!

How much square footage was reduced at the Pentagon?

Shawn: We were somewhere between 100,000-140,000 square feet. But was brought down to 3,500-3,600 square feet.

The only piece to virtualization that I see missing is what to do when you have different types of software on a server that don’t work together. How do you manage that?

Teresa: We used to require separate servers for each component - servers, applications, etc. It was a 1-1 ratio. Now we are at 12-1 or as high as 40-1.

Is this a situation achieved through VMWare or are there other alternatives?

Bajinder: There are other alternatives, but VMWare was the predominant vendor at OCC, we achieved a 9-1 consolidation.

What host OS are you running on?

Bajinder: Windows, not Linux. We contemplated Linux.

Shawn: Pentagon is running Linux and Solaris86. We run everything.

How do you overcome security challenges of going virtual?

Virtualization is all about change management. It impacts how we manage resources. The bottom line is it’s policies on how to manage it, and a discipline. We have to micromanage.

Questions about CAC card access

Virtualization doesn’t change the application security environment.

The cost for virtualization seem substantial when you talk about software and hardware. Was there a net savings?

Shawn: Virtualization is a challenge because if you’re just trying to measure yourself on consolidation, you may never get there with ROI. Unless you change the way you conduct business, and get business impacts - R&D to production, etc. - all these things you’re not getting today. VMWare, virtualization, Hyper-Visor DOES add complexity to the environment. You have to make an executive decision to change the way you do your business. All of it is hard to sell, but the organizational change (ITIL, SOA, etc.) - you have a better IT infrastructure to support your business.

Bajinder: When you’re examining close to 7 trillion in assets, the ability to manage risk is the ROI.

Teresa: It is hard to articulate the value, but important to do so.

Is licensing a challenge?

Shawn: Yes. We found that we would build that into the cost for the datacenter. A big high-end system can run a lot of applications.

Should you buy more rack machines? Or blades?

Shawn: It depends. We do both. It depends on mission critical applications. Sometimes it makes sense to have a two processor system.

As far as governments and policy, how did you get the buy-in to consolidate servers?

Bajinder: OCC has one infrastructure, so we didn’t have that issue.

Shawn: That was a significant challenge. There were challenges with legacy stuff. Part of our consolidation was to build a housing area for those legacies.

Are there any standards around virtualization?

Shawn: It’s concerning that there is one vendor that so dominates the virtualization market right now. We’d like to see someone else pop up, but VMWare is so far ahead of everyone else.

We saved the best for last, when I asked:

What about tools? Presumably you have a management or monitoring infrastructure in place. Did you have to rip things out that didn’t work with virtualization?

Shawn: Most of our tools monitoring was very standardized. No customization. We’re looking at BladeLogic from VMWare (actually, they just got bought by BMC) for change management. We want to make sure we have the granularity to handle access controls. We want to be able to handle the complexity. Change management is the biggest issue.

Bajinder: We depend on people to maintain a standardized environment. As we introduce virtualization, we want to maintain a cost of ownership.

Popularity: 78% [?]

April 2nd, 2008