Intuition is under fire
Two months ago I saw a television commercial for Business Intelligence software, and in the commercial a customer being interviewed had the following to say:
“We used to use intuition; now we use analytics.”
In other words, we’re being asked to believe that the business owner was able to make progress by replacing decision-making using intuition with decision-making using analytics.
The statement didn’t sit well with me, so I immediately tweeted the following:
IBM ad: "We used to use intuition; now we use analytics." <- this is the wrong mindset. One should complement, not replace, the other.
— Ben Jones (@DataRemixed) April 13, 2014
Working in the BI industry, I’ve heard similar attacks on human intuition from many different sides, and I don’t agree with it at all. I had a chance to say more about my objection to this notion at the eyeo festival last week in a brief talk entitled “Intuition Still Matters: Using Tableau to Think with Your Data”. You can see the slides to this presentation here.
In this blog post I’d like to explain why I feel that in a world awash with data, human intuition is actually more valuable than ever. Briefly, human intuition is the spark plug that makes the analytics engine run.
Intuition wasn’t always a byword
Contrast the commercial’s negative attitude toward human intuition with Albert Einstein’s rather glowing appraisal:
Without a doubt, it would be hard to come up with a more positive statement about intuition than this. So which is it? Is human intuition a faulty and antiquated decision-making tool, in dire need of replacement with something better, or is it the only valuable thing there is?
Before we go any further, we should define the terms.
What is intuition?
Oxford dictionary defines intuition as follows:
“The ability to understand something immediately, without the need for conscious reasoning”
It comes from the Latin root word intuērī, mean to look at or gaze upon. Thus the etymology of the word links it with the human visual system. Sight and intuition both occur instantaneously and effortlessly. Both can also mislead, of which more later. With intuition as with sight, the awareness comes before any logical explanation.
The link between intuition and sight is often a very literal one. In social situations, we intuitively sense other people’s emotions when we first lay eyes on their facial expressions:
Fig 1. Basic facial expressions (source)
And with abstract representations of data, we spot the marks that have certain unusual attributes in an intuitive way – that is, we notice them without having to think about it. We call these attributes “preattentive”. Noticing them doesn’t take effort – it’s as if it happens to us. Here are two examples: what do you immediately notice about them?
Fig 2. Two preattentive attributes (source)
Similarly, we can feel a compelling sense of confidence about what’s going to happen in the future, and what we should do about it. This is what is commonly meant when someone says a person has a great intuition about a specific field.
Intuition is commonly contrasted with reason, “the power of the mind to think, understand, and form judgments by a process of logic.” Logic, in turn, involves “strict principles of validity”. And analytics is “information resulting from the systematic analysis of data or statistics.”
To make the best decisions in business and in life, we need to be adept at many different forms of thinking, including intuition, and we need to know how to incorporate many different types of inputs, including numerical data and statistics (analytics). Intuition and analytics don’t have to be seen as mutually exclusive at all. In fact, they can be viewed as complementary.
Let me give some examples of how intuition provides the spark for the analytical process.
Five Reasons Why Intuition Still Matters
- Knowing what to measure in the first place
Any process has an almost infinite number of variables that could be tracked and analyzed. On which should we spend our time? Knowing where to start can be a problem, especially if we know very little about the subject we’re dealing with.
One school of thought goes something like this: collect data on everything and let an algorithm tell you which to pay attention to.
Sorry, I don’t buy it.
- First, not even the NSA collects data on “everything”. I guarantee you a filter has been applied to narrow the set of inputs. God may have counted every hair on your head, but I seriously doubt anyone else has.
- Second, while data mining algorithms can discover notable patterns in huge data sets, only human intuition can discern between the useful patterns and the useless ones. They get their very “usefulness” from our goals and values.
Once we pick data to collect and metrics to analyze, what do the numbers tell us? We talked about preattentive attributes briefly – via data visualization, our intuition can be put to good use interpreting the important 1’s and 0’s in the databases we’ve meticulously built.
Using intuition in this way isn’t a perfect process though. Just as we might recoil from a garden hose that our instincts tell us is a snake, we can see signals in data that aren’t really there. Alternately, we can miss really important signals that are there. Just because intuition doesn’t work perfectly, though, doesn’t mean it should be discarded. We just need to hone our intuition for working with numbers, and we need to distrust it somewhat.
Jonas Salk, the American medical researcher who developed the first polio vaccine, had the following to say about intuition in his book Anatomy of Reality: Merging of Intuition and Reason:
Fig 3. From Jonas Salk’s Anatomy of Reality: Merging of Intuition and Reason
He made a discovery that has saved the lives of countless people in the world, and he chalked up an important part of his success to intuition. Often the best outcome of an interaction with data is that we sense another, even better question to ask. And the process iterates. The realization of the next place to look can form in our mind like an intuitive spark. The light bulb analogy applies.
For many types of questions or problems, we could continue to search for a solution ad nauseum. Think of a chess game. What’s the “best move” to make at a given point in the game? Russian chess Grandmaster Garry Kasparov knew something about this question, and here’s how he understood it, as stated in his book How Life Imitates Chess:
Fig 4. From Kasparov’s How Life Imitates Chess
There comes a point in time when it’s best to stop analyzing and make a move. Knowing when we’ve arrived at this point is a function of intuition. If we don’t have this intuitive switch, we can suffer from “analysis paralysis”, and then we go nowhere. We’ve all been there.
A key part of the data discovery process is communicating our findings with others. We can use our intuition to choose the best message, channel, venue, visualization types, aesthetic elements, timing, tone, pace, etc. If we have a deep understanding of our audience, we will intuitively know what will get through to them, and what will fall on deaf ears. When we get it right, it can be a wonder to behold. Think Hans Rosling.
Crafting the communication is a creative process, and human intuition will need to be tapped to do it well.
For the reasons outlined above, I don’t believe that human intuition will ever be rendered obsolete. No matter how smart our algorithms get, no matter how sophisticated our tools or methods, the intuitive “spark” in the human mind will always be the key element in our thoughts, in our decisions and in our discoveries. Data and analytics can fuel these sparks, and they can provide a way to make sure we’re headed in the right direction, but they can’t replace human intuition. Not the way I understand it, anyway.
I don’t think the creators of the commercial would disagree with this point of view, so it likely comes down to semantics. Maybe the business owner in the commercial should have said: “We used to rely on intuition alone, now we combine it with analytics to make even better decisions.” Slightly less snappy, I know. But at least intuition doesn’t get thrown under the bus.