David Menninger's Analyst Perspectives

In various forms, business intelligence (BI) – as queries, reporting, dashboards and online analytical processing (OLAP) – is being used increasingly widely. And as basic BI capabilities spread to more organizations, innovative ones increasingly are exploring how to take advantage of the next step in the strategic use of BI: predictive analytics. The trend in Web searches for the phrase “predictive analytics” gives one indication of the rise in interest in this area. From 2004 through 2008, the number of Web search was relatively steady. Beginning in 2009, the number of searches rose significantly and has continued to rise.

While a market for products that deliver predictive analytics has existed for many years and enterprises in a few vertical industries and specific lines of business have been willing to pay large sums to acquire those capabilities, this constitutes but a fraction of the potential user base. Predictive analytics are nowhere near the top of the list of requirements for business intelligence implementations. In our recent benchmark research on business analytics, participants from more than 2,600 organizations ranked predictive analytics 10th among technologies they use today to generate analytics; only one in eight companies use them. Planning and forecasting fared only slightly better, ranking sixth with use by 26 percent. Yet in other research, 80 percent of organizations indicated that applying predictive analytics is important or very important.

For the untapped majority, technology has now advanced to a stage at which it is feasible to harness the potential of predictive analytics, and the value of these advanced analytics is increasingly clear. Like BI, predictive analytics also can appear in several identities, including predictive statistics, forecasting, simulation, optimization and data mining. Among lines of business, contact centers and supply chains are most interested in predictive analytics. By industry, telecommunications, medicine and financial services are among the heavier users of these analytics, but even there no more than 20 percent now employ them. Ironically, our recent research on analytics shows that finance departments are the line of business least likely to use predictive analytics, even though they could find wide applicability for them.

Predictive analytics defined broadly have a checkered history. Some software vendors have long offered tools for statistics, forecasting and data mining including Angoss, SAS and SPSS, which is now part of IBM. I was part of the team at Oracle that acquired data mining technology for the Oracle database a dozen years ago. Companies like KXEN sprang up around that time, perhaps hoping to capitalize on what appeared to be a growing market. The open source statistics project known simply as R has been around for more than a decade and some BI vendors like Information Builders have embedded into their product called WebFOCUS RStat.

But then things quieted down for almost a decade as the large vendors focused on consolidating BI capabilities into a single platform. Around 2007, interest began to rise again, as BI vendors looked for other ways to differentiate their product offerings. Information Builders began touting its successes in predictive analytics with a customer case study of how the Richmond, Va., police department was using the technology to fight crime. In 2008 Tibco acquired Insightful and Netezza acquired NuTech, both to gain more advanced analytics. And in 2009 IBM acquired SPSS.

As well these combinations, there are signs that more vendors see a market in supplying advanced analytics. Several startups have focused their attention on predictive analytics, including some such as Predixion with offerings that run in the cloud. Database companies have been pushing to add more advanced analytics to their products, some by forming partnerships with or acquiring advanced analytics vendors and others by building their own analytics frameworks. The trend has even been noted by mainstream media: For example, in 2009 an article in The New York Times suggested that “data mining has entered a golden age, whether being used to set ad prices, find new drugs more quickly or fine-tune financial models.”

Thus the predictive analytics market seems to be coming to life just as need intensifies. The huge volumes of data processed by major websites and online retailers, to name just two types, cannot be moved easily or quickly to a separate system for more advanced analyses. The Apache Hadoop project has been extended with a project called Mahout, a machine learning library that supports large data sets. Simultaneously, data warehouse vendors have been adding more analytics to their databases. Other vendors are touting in-memory systems as the way to deliver advanced analytics on large-scale data. It is possible that this marriage of analytics with large-scale data will cause a surge in the use of predictive analytics – unless other challenges prevent more widespread use of these capabilities.

One is the inherent complexity of the analytics themselves. They employ arcane algorithms and often require knowledge of sampling techniques to avoid bias in the analyses. Developing such analytics tools involves creating models, deploying them and managing them to understand when a model has become “stale” or outdated and ought to be revised or replaced. It should be obvious that only the most technically advanced user will have (or desire) any familiarity with this complexity; to achieve broad adoption, either there must be more highly trained users (which is unlikely) or vendors must mask the complexity and make their tools easier to use.

It also is possible that a lack of understanding of the business value these analyses provide is holding back adoption. Predictive analytics have been employed to advantage in certain segments of the market; other segments can learn from those deployments, but it is up to vendors to make the benefits and examples more widely known. In an upcoming benchmark research I’ll be investigating what is necessary to deploy these techniques more broadly and looking for the best practices from successful implementations.

Regards,

David Menninger – VP & Research Director