The Big Data 'Opportunity Area' for Innovation

Why Big Data is Not the Answer for Innovation

Recently, at the excellent 2014 MBA Innovation Summit put on by Columbia, Yale and Wharton, I listened to Ron Gonen, NYC’s Deputy Commissioner of Sanitation, talk about how his organization is using big data to address big problems faced by the city.

By collecting tons of analytical data, the city is now able to identify everything from the illegal subdivisions (where landlords have divided apartments into too many rooms for fire capacity) that are most at risk for deadly fires to areas most attractive to entrepreneurs.  NYC is using that data to identify trends and to figure out the best ways to allocate their scarce resources in order to save lives.

Seeing the trends and insights revealed by big data is awe-inspiring. And I’m not alone in thinking so. A 2013 survey of senior corporate executives from Fortune 1000 companies (pdf) revealed that 60% have implemented at least one big data initiative, with 32% reporting that they’ve implemented big data across the organization.  Another 31% have initiatives on the horizon.  Only 9% of executives aren’t forging ahead into using big data.

Coming from Nowhere?

Why has big data come out of nowhere like this?  The 2012 election threw it into national attention as after Barack Obama’s team won by investing massive amounts of time and energy micro-targeting voters based on their likes, habits, behaviors, etc. His campaign turned out voters who organizers didn’t even know existed! After that it was a short leap for the business world to see the potential of big data in helping them understand their customers.

Improving Customer Experience

Now, organizations are using big data to analyze all the ways that their customers interact with them.  85% of executives reported in the survey that they’re investing in big data initiatives to gain new insights about their customers.  They’re looking to improve their fact-based decision-making, discover new patterns, and figure out ways to improve the customer experience and to create new product innovations.

Jim Smith, head of Wells Fargo’s Enterprise Data and Analytics and Digital Channels groups says, “Our customers interact with us in many different channels and there has been tremendous data growth with the surge of online and mobile banking. Each of these interactions provides us with an opportunity to more accurately identify a customer’s specific needs and interests.  From there, we can evolve or improve how we provide a service or develop a new one.”

Math Trumping Science

Big data’s appeal to executives is obvious because it predicts how every action or initiative will turn out based on what has happened before. As said by Vivek Ranadivé, founder and CEO of financial-data software provider TIBCO: “I believe that math is trumping science. What I mean by that is you don’t really have to know why, you just have to know that if A and B happen, C will happen.” This logic is why 64% of executives cite “new product development and innovation” as one of their main reasons for investing in big data (pdf).

Big Data’s Problem

But investing in big data isn’t enough to result in innovation. A recent survey by IDG Research Services and Kapow Software of more than 200 IT and business leaders at large organizations revealed that just 23% of leaders think their big data projects have been a success. Only 52% think their big data projects have been “somewhat successful”.

So what’s going on? Why the dissatisfaction with how big data is helping companies innovate?

The problem with Big Data is how people assume that the solutions are all buried in the data – and that all it takes it to analyze it in order to reveal new possibilities. That’s simply not how new dramatic invention happens.

Big data would have suggested that the best way to improve transportation speed would have been to refine the bloodlines of horses – not to invent trains, planes or automobiles. Heck, even riding horses was an innovation that could not have been predicted by analyzing the foot transportation of prehistoric humans! In each of those world-changing inventions, more information about the existing world would have been useless in predicting the ideas that changed it.

Opportunity Areas for Innovation

Where big data offers great value, however, is in revealing “Opportunity Areas” for innovation. If you know the data about where and when landlords are creating illegal (and dangerous) subdivisions of apartments in NYC, you can come up with new ideas for how to combat it. If you know that your customers are really seeking to get from Point A to Point B faster, you can explore ways of solving that need.

Big data gives companies the ability to compare how their consumers act with what they say matters to them – shedding insight on how, why and when they purchase.  That knowledge gives executives the ability to  generate new product ideas that will really hit their consumers’ unmet needs and match with how they purchase.

Big data is full of temptation because it feels like – if you just get more information – you’ll be able to find the hidden answer.  That’s what’s drawing so many executives in.  But where big data really offers the opportunity to bring value is in uncovering unknown insights into how consumers really act. Smart executives will then use the insights that they’ve uncovered as “Opportunity Areas” that they’ll focus their innovation efforts within.

image credit: stampede-design.com