Discover Performance

HP Software's community for IT leaders // July 2014

Big Data in 2020: Making the patterns emerge

In the next decade, predictive monitoring will be powerful and intuitive. That’s our prediction—and HP Labs’ goal.

The bottom line

What: A key component of the analytics future is the ability to have software monitor—and flag—anomalous patterns in complex systems.
Why: Complexity, tedium, and humans’ relative slowness mean we need better tools.
How: HP Labs is working on a vision for predictive monitoring.
More: Download “Big Data 20/20” from our Enterprise 20/20 page.

By Mike Shaw, HP Software Strategic Marketing

Normal and abnormal patterns: we humans need to be good at telling the difference between the two to survive and thrive. Is the stranger walking toward us a threat? Is the food we’re eating normal, or might it make us ill? Is that twinge in our shoulder or shortness of breath a sign of abnormal health? Are the other drivers on the road with us behaving dangerously?

Mike Shaw
At work, recognizing normal and abnormal behaviors is important too. Retailers react to changing sales patterns, banks and insurers keep an eye out for fraud, and doctors look for signs of illness. IT departments have to be alert for operational crises and subtle security threats. In other words, humans are constantly scanning the systems with which they interact, looking for the emergence of abnormal behavior patterns.

‘What’s wrong with this picture?’

Big Data analytics will augment humans in numerous ways, from making robust analytics tools more widely available to prompting us with information and recommendations based on our current situation, goals, and preferences. Advanced analytics will also help humans to quickly recognize the onset of abnormal behavior patterns in complex systems.

Why do humans need help to predict the onset of abnormal behaviors? Three reasons:

  1. The systems that we create are becoming ever more complex. We’re great at monitoring one variable, but poor at tracking several at once.
  2. Monitoring is boring. Freeing the human brain from boring work lets us focus on what computers can’t do.
  3. Computers can spot abnormal behavior faster than we can. Also, their ability to monitor many variables at the same time means that they can predict an impending anomaly. We can then proactively adjust so the anomaly never even emerges.

HP Labs and others are researching the use of Big Data techniques to predict abnormal behaviors. Such "anomaly prediction systems" will work something like this:

1. Specify the data feeds. A subject matter expert (a marketing manager, a specialist doctor, a city transportation system expert, a security expert) will tell the system what data feeds to monitor. The marketing team for a fast-moving consumer goods retailer might have a system that monitors sales over time, geography, direct/indirect mix, weather, time of year, and pricing versus the competition.

2. Model normal and anomaly patterns. The prediction system will take a guess at which patterns are normal versus anomalous. The subject matter expert will then check the system’s work and reclassify patterns as needed—rather than having had to enter hundreds of rules up front.

3. Monitor and predict. Constantly monitoring for variations in normal patterns, the system will flag the right expert when an anomaly starts to emerge. The expert will interrogate the prediction system’s logic to understand why an anomaly is flagged—and if it’s a "false positive," the system will learn more about the normal/anomalous models. 

Even better, says Dr. Renato Keshet, the research manager of the analytics research lab at HP Labs, Israel, is the likelihood that the prediction system’s initial analysis will throw up relationships the expert hadn’t seen before.

"When humans need to look at data, we typically use a tool like Microsoft Excel and we plot one variable against another," Keshet says. A European marketing manager might notice anomalous sales patterns when looking by country. "But the prediction system might notice a stronger relationship between whether a region uses resellers or goes direct than there is between countries and anomalies."

Putting prediction technology to work

Let’s look at how this prediction technology might be used:

  • HP Labs is working on improving IT operations’ ability to predict and proactively avoid application performance problems—before customers notice them. 
  • The same HP Labs team is applying the technology to IT security, to spot well-hidden adversaries.
  • Marketing teams will better monitor the sales of fast-moving consumer goods. 
  • Doctors, monitoring patients at home and in hospitals, can more quickly learn of health-threatening anomalies. 
  • Feeds from a city’s transportation systems can better manage—and prevent—traffic jams.

Eye on the future

The ability to monitor complex systems and predict anomalies is another example of how Big Data analytics will increasingly augment human capabilities. Enabling subject matter experts to do the modeling themselves is a key to the adoption and success of such systems. As we move toward 2020 and beyond, this democratization of real analytics power will be revolutionary.

Mike Shaw’s recent columns on Big Data form the core of the new Enterprise 20/20 chapter, "Big Data 20/20," looking at the transformative future of advanced analytics. Download the complete chapter—and find previous chapters—on our Enterprise 20/20 page.


IT leader assessment

This tool evaluates the correlation between IT attributes and business success and, based on how your answers compare with average scores, will advise you where to invest in IT.

It is based on data HP collected from 650 global companies about a range of IT characteristics (server capacities, approach to information management, security, BYOD, etc.) and how they correlate to revenue gain. This assessment will compare your answers to the average scores in that study.

There are 12 questions that will require an estimated 10 minutes of your time. You'll receive a summary of your rating upon completion.

Let's get started

Please select an answer.


Your answer:
Your score:
Average score:
Revenue leaders' score:


Please select an answer.



Your score:
Average score:
Revenue leaders' score:

Get detailed results:


Popular tags


Discover Performance Weekly

HP Software’s Paul Muller hosts a weekly video digging into the hottest IT issues. Check out the latest episodes.

DevOps: Teams, not silos

Discover Performance’s Paul Muller and DevOps guru Damon Edwards discuss what DevOps brings to a business’s bottom line, and how to get your people out of their silos. Feb. 24.

Enterprise 20/20

Introduction to Enterprise 20/20

What will a successful enterprise look like in the future?

CIO 20/20

Challenges and opportunities for the CIO of the future.

Employee 20/20

What the workforce of 2020 can expect from IT, and what IT can expect from the workforce.

Data Center 20/20

The innovation and revenue engine of the enterprise.

Dev Center 20/20

How will we organize development centers for the apps that will power our enterprises?

Marketing 20/20

Welcome to a new reality of split-second decisions and marketing by the numbers.

IT Operations 20/20

How can you achieve the data center of the future?

Security 20/20

Preparing today for tomorrow’s threats.

Mobility 20/20

Looking toward the era when everyone — and everything — is connected.

Read more

HP Software related

Most read articles

Discover Performance


Tweets @ HPITperformance