CIOIN

For too long now innovation has been viewed as a black art. Business journals, such as Business Week and Fortune, that regularly publish their lists of the most innovative companies, select top performers based on surveys of CEO/business executives, not hard data of performances. Similarly, few managers have the required metrics to make informed decisions about their innovation programs. Therefore, managers of all types, and IT managers especially (since they are often out of the strategy decision-making loop), have only a vague sense of the innovativeness of their company and their department; they have little or no means to assess the effectiveness and efficiency of a particular innovation program. Over the last decade, however, many organizations and their IT departments, have successfully implemented enterprise-wide as well as localized innovation programs. Along with their successes, and some pioneers' failures, we now have a more thorough understanding of what and how to measure an organization's innovation program.

What to Measure?

Ultimately, innovation is a means to an end -- a competence for generating profitable growth opportunities and improving the organization's competitiveness. A a holistic measurement system needs to have three perspectives: performance, strength of the competence, and strategic application of the competence. The performance perspectives report out the "returns" or "results" of an organization's innovation program(s) while the competence perspectives report out the ability to envision and implement innovative opportunities. The strategy perspective outlines the criticality and impact of innovation in the organization's strategic direction.

Performance Perspective

To measure the performance, most companies simply track the revenues and profit contributions of their new products and services. Given the flexibility and capabilities of today's ERP and/or financial management systems, tracking investment, direct cost, direct revenues, profit margins and other revenues and profit measures is not highly complicated, as long as the specific products or services can be labeled appropriately. The same is fairly true for IT organizations. While it's hard to quantify the total value of the IT department with any precision, it's substantially more feasible to value the cost and benefit of specific new programs and initiatives.

Additionally, given the lag time between the envisioning of innovative ideas and commercialization of these ideas, it's important to measure one's innovation pipeline. Specifically, managers should track the value and status of the innovation pipeline.

One way to estimate the value of pipeline is to sum the estimated net present values of all the opportunities in the pipeline, in a similar way as pharmaceutical companies estimate and track the value of drugs in various stages of development. Specifically, the value of the "innovation pipeline" is the sum of the values of opportunities, each then discounted by the probability of getting to market and time-value, for example:

VALUE = NPV of Opportunity 1 x Probability of getting to market (based on the stage it's in)
+ NPV of Opportunity 2 x Probability of getting to market (based on the stage it's in)
+ NPV of Opportunity 3 x Probability of getting to market (based on the stage it's in)
& + NPV of Opportunity N x Probability of getting to market (based on the stage it's in)

A few years ago, a major global commercial bank began an innovation program in its IT organization where employees around the globe share and nominate improvement opportunities. These opportunities were elaborated and evaluated by the online communities, and the most attractive ones were then adopted by regional operational units. About a year into the program the bank was able to quantify the value of output by assessing the benefit of each opportunity implemented. Then, it revised the opportunity submission form to include an estimation of benefits (in magnitude, not a precise number), and this benefit estimation was further refined as the opportunity went through online elaboration/evaluation and implementation. Now, the bank's IT organization reports to its management team the value of its innovation project portfolio and results.

The valuation process of the IT innovation pipeline is similar to the valuation of a new product pipeline, although this bank's IT innovation pipeline has fewer stages than the typical new product development pipeline, and the benefit/cost valuation is based on estimation, not a detailed accounting of actual revenue generated and cost incurred.

Once this calculation is standardized, the change in the total-value of pipeline can be calculated for each time period. Then, the ratio between this number and the investment in innovation for the same time period is a proxy for return on investment for innovation (innovation ROI). An implication is that the value of the pipeline and ROI can be increased by adding attractive opportunities in addition to moving opportunities through the pipeline.

Similarly, at any one time, the number of opportunities at each stage of development is a quick overview for the health and growth of the pipeline.

Together, overall revenue/profit contribution from innovation, value of innovation pipeline, status of innovation pipeline and innovation ROI can offer management a fairly robust view of innovation performance.

Competence Perspective

Competence Perspective

Since innovation is a means to an end, savvy CIOs and IT managers go beyond measuring the results of innovation. They also assess the strength or capacity of this competence. This perspective assesses the extent to which the organization's skills, processes, culture, and conditions support the conversion of innovation resources into opportunities for business renewal.

The inputs of this perspective are the preconditions for innovation, i.e. the extent to which an organization's skills, tools, culture, and values are adapted to innovation. For example, organizations can assess employees' personal competence levels, calculate the numbers of employees at various performance levels, measure employees' access to innovation training tools, and benchmark the breadth and quality of training/professional development programs.

The codification of the capability measures the degree to which this capability is embedded and standardized. While the innovation activities will always rely on the skills and knowledge of the practitioners, many of the methods and techniques can be codified, instead of reinvented at each iteration. In this area, organizations can measure the extent to which the business and application implementation processes are codified and embedded, and the level of automation/information system support available.

The outputs of the capability view measure the organization's success at providing new technology-centric capabilities to the enterprise. For example, it might measure new competencies (i.e. proficiency with a new technology, adoption of a new application development approach) or newly created strategic applications (i.e., new pricing application, new supply chain management system). As with the resource view, measurement of both inputs and outputs is necessary to monitor the extent to which capability view inputs seem to drive capability view outputs.

The input, codification, and output measures should offer management a succinct perspective of an organization's competence strength in innovation.

Strategy Perspective

This perspective assesses the degree of strategic importance of innovation in the entire organization and IT for the overall organization. It uses measures such as company's leadership involvement in innovation activities, the establishment of formal processes to promote innovation, extent of innovation is embedded into performance metrics, and the criticality of IT in supporting the company's revenue and profit growth.

Three Common Pitfalls of Innovation Measurements

As more and more organizations embrace innovation as a critical organizational capability, many have also gained operational experiences at measuring innovation. Amongst these pioneers, they have experienced the following common mistakes:

Setting arbitrary high targets or hurdles for innovation projects. While many organizations have found their innovative products/services performed better--in terms of profitability or growth--than their traditional products/services, to arbitrarily set higher hurdles or performance targets could unnecessarily exclude a set of high potential opportunities. For example, one U.S.-based financial services/insurance company considered using a higher cost of capital for evaluating innovation projects. While it was correct to assume more unknowns or obstacles for innovation projects, these knowledge gaps didn't automatically translate into financial risks. Under such a high cost of capital constraint, the company would not have pursued an innovative opportunity that attracted consumers with small initial deposits but then actively grew these accounts over a long horizon. Given that this offering had been based on existing products and services, the financial risks were already well analyzed and mitigated so that, subjecting this opportunity to a higher cost of capital would have caused this company to forgo its now successful new business.

Implication for IT: When implementing a new technology or applying a new application development approach there are many unknowns. So, the budgeting process should create the appropriate buffer (time, cost) to account for the unknowns. Then, setting the project with higher hurdle would be "double-counting" the risk.

Placing too much emphasis on any one measure. The three perspectives and a range of measures offer a holistic picture of innovation for managers, and placing too much emphasis on just one or a few of the measures could lead to unintended consequences. For example, one medical equipment manufacturer focused on getting as many employees as possible trained on innovation, but the lack of depth by any trainees kept the organization from fundamentally altering its culture, business processes, incentive system and other enablers of an innovation system. Another organization measured every employee's innovation revenue contribution, and this diverted a high amount of resource on debating if commercialized products/services were innovation. The organization would have been better served if that same resource was applied to improving existing products/services or envisioning/developing novel products/services.

Implication for IT: Just like enterprises need to apply the entire spectrum of innovation measures in managing its innovation program, IT managers and executives should utilize the entire spectrum as well.

Separating measures from business decisions. Years after a consumer product manufacturer began embedding innovation across its entire organization, it successfully trained a critical mass of innovation practitioners and constructed a robust pipeline of innovation opportunities. Along the way, this company automated tracking of revenues and profits from innovation, and codified a range of business processes. Then, it decided to delegate the decision of "tagging" innovation to products/services in its ERP to a committee of middle managers. The result was diverting resources from applying innovation thinking to improve products/services ex ante to debating if specific products/services met the innovation definition ex post of the decision to pursue these opportunities.

Implication for IT: While estimating benefits and quantifying the value of innovative programs/initiatives help IT and enterprise executive make informed business decision. But, ultimately it's a business decision; so, don't let the desire to archive some performance target supersede business sense.

Smart Metrics for Better Decisions

Innovation is a means to an end. Therefore, organizations should measure across the perspectives of performance, competence and strategy. These three perspectives help managers understand the results, the capacity for performance, and application of innovation. However, don't assume a sophisticated or comprehensive measurement system replaces the need for management deliberation and decision-making. Successful innovation pioneers develop their innovation measurement system in parallel with management and business decision processes that take advantage of the rich information. In the end, it's not just about better knowledge -- it's about better decisions enabled by better knowledge.