Discover PerformanceHP Software's community for IT leaders // July 2014
Big Data in 2020: Making the patterns emerge
In the next decade, predictive monitoring will be powerful and intuitive. That’s our prediction—and HP Labs’ goal.
By Mike Shaw, HP Software Strategic Marketing
Normal and abnormal patterns: we humans need to be good at telling the difference between the two to survive and thrive. Is the stranger walking toward us a threat? Is the food we’re eating normal, or might it make us ill? Is that twinge in our shoulder or shortness of breath a sign of abnormal health? Are the other drivers on the road with us behaving dangerously?
At work, recognizing normal and abnormal behaviors is important too. Retailers react to changing sales patterns, banks and insurers keep an eye out for fraud, and doctors look for signs of illness. IT departments have to be alert for operational crises and subtle security threats. In other words, humans are constantly scanning the systems with which they interact, looking for the emergence of abnormal behavior patterns.
‘What’s wrong with this picture?’
Big Data analytics will augment humans in numerous ways, from making robust analytics tools more widely available to prompting us with information and recommendations based on our current situation, goals, and preferences. Advanced analytics will also help humans to quickly recognize the onset of abnormal behavior patterns in complex systems.
Why do humans need help to predict the onset of abnormal behaviors? Three reasons:
- The systems that we create are becoming ever more complex. We’re great at monitoring one variable, but poor at tracking several at once.
- Monitoring is boring. Freeing the human brain from boring work lets us focus on what computers can’t do.
- Computers can spot abnormal behavior faster than we can. Also, their ability to monitor many variables at the same time means that they can predict an impending anomaly. We can then proactively adjust so the anomaly never even emerges.
HP Labs and others are researching the use of Big Data techniques to predict abnormal behaviors. Such "anomaly prediction systems" will work something like this:
1. Specify the data feeds. A subject matter expert (a marketing manager, a specialist doctor, a city transportation system expert, a security expert) will tell the system what data feeds to monitor. The marketing team for a fast-moving consumer goods retailer might have a system that monitors sales over time, geography, direct/indirect mix, weather, time of year, and pricing versus the competition.
2. Model normal and anomaly patterns. The prediction system will take a guess at which patterns are normal versus anomalous. The subject matter expert will then check the system’s work and reclassify patterns as needed—rather than having had to enter hundreds of rules up front.
3. Monitor and predict. Constantly monitoring for variations in normal patterns, the system will flag the right expert when an anomaly starts to emerge. The expert will interrogate the prediction system’s logic to understand why an anomaly is flagged—and if it’s a "false positive," the system will learn more about the normal/anomalous models.
Even better, says Dr. Renato Keshet, the research manager of the analytics research lab at HP Labs, Israel, is the likelihood that the prediction system’s initial analysis will throw up relationships the expert hadn’t seen before.
"When humans need to look at data, we typically use a tool like Microsoft Excel and we plot one variable against another," Keshet says. A European marketing manager might notice anomalous sales patterns when looking by country. "But the prediction system might notice a stronger relationship between whether a region uses resellers or goes direct than there is between countries and anomalies."
Putting prediction technology to work
Let’s look at how this prediction technology might be used:
- HP Labs is working on improving IT operations’ ability to predict and proactively avoid application performance problems—before customers notice them.
- The same HP Labs team is applying the technology to IT security, to spot well-hidden adversaries.
- Marketing teams will better monitor the sales of fast-moving consumer goods.
- Doctors, monitoring patients at home and in hospitals, can more quickly learn of health-threatening anomalies.
- Feeds from a city’s transportation systems can better manage—and prevent—traffic jams.
Eye on the future
The ability to monitor complex systems and predict anomalies is another example of how Big Data analytics will increasingly augment human capabilities. Enabling subject matter experts to do the modeling themselves is a key to the adoption and success of such systems. As we move toward 2020 and beyond, this democratization of real analytics power will be revolutionary.
Mike Shaw’s recent columns on Big Data form the core of the new Enterprise 20/20 chapter, "Big Data 20/20," looking at the transformative future of advanced analytics. Download the complete chapter—and find previous chapters—on our Enterprise 20/20 page.
HP Software’s Paul Muller hosts a weekly video digging into the hottest IT issues. Check out the latest episodes.
Introduction to Enterprise 20/20
What will a successful enterprise look like in the future?
Challenges and opportunities for the CIO of the future.
What the workforce of 2020 can expect from IT, and what IT can expect from the workforce.
Data Center 20/20
The innovation and revenue engine of the enterprise.
Dev Center 20/20
How will we organize development centers for the apps that will power our enterprises?
Welcome to a new reality of split-second decisions and marketing by the numbers.
IT Operations 20/20
How can you achieve the data center of the future?
Preparing today for tomorrow’s threats.
Looking toward the era when everyone — and everything — is connected.