Intuition is under fire
Two months ago I saw a television commercial for Business Intelligence software, and in the commercial a customer being interviewed had the following to say:
“We used to use intuition; now we use analytics.”
In other words, we’re being asked to believe that the business owner was able to make progress by replacing decision-making using intuition with decision-making using analytics.
The statement didn’t sit well with me, so I immediately tweeted the following:
IBM ad: "We used to use intuition; now we use analytics." <- this is the wrong mindset. One should complement, not replace, the other.
— Ben Jones (@DataRemixed) April 13, 2014
Working in the BI industry, I’ve heard similar attacks on human intuition from many different sides, and I don’t agree with it at all. I had a chance to say more about my objection to this notion at the eyeo festival last week in a brief talk entitled “Intuition Still Matters: Using Tableau to Think with Your Data”. You can see the slides to this presentation here.
In this blog post I’d like to explain why I feel that in a world awash with data, human intuition is actually more valuable than ever. Briefly, human intuition is the spark plug that makes the analytics engine run.
Intuition wasn’t always a byword
Contrast the commercial’s negative attitude toward human intuition with Albert Einstein’s rather glowing appraisal:
Without a doubt, it would be hard to come up with a more positive statement about intuition than this. So which is it? Is human intuition a faulty and antiquated decision-making tool, in dire need of replacement with something better, or is it the only valuable thing there is?
Before we go any further, we should define the terms.
What is intuition?
Oxford dictionary defines intuition as follows:
“The ability to understand something immediately, without the need for conscious reasoning”
It comes from the Latin root word intuērī, mean to look at or gaze upon. Thus the etymology of the word links it with the human visual system. Sight and intuition both occur instantaneously and effortlessly. Both can also mislead, of which more later. With intuition as with sight, the awareness comes before any logical explanation.
The link between intuition and sight is often a very literal one. In social situations, we intuitively sense other people’s emotions when we first lay eyes on their facial expressions:
Fig 1. Basic facial expressions (source)
And with abstract representations of data, we spot the marks that have certain unusual attributes in an intuitive way – that is, we notice them without having to think about it. We call these attributes “preattentive”. Noticing them doesn’t take effort – it’s as if it happens to us. Here are two examples: what do you immediately notice about them?
Fig 2. Two preattentive attributes (source)
Similarly, we can feel a compelling sense of confidence about what’s going to happen in the future, and what we should do about it. This is what is commonly meant when someone says a person has a great intuition about a specific field.
Intuition is commonly contrasted with reason, “the power of the mind to think, understand, and form judgments by a process of logic.” Logic, in turn, involves “strict principles of validity”. And analytics is “information resulting from the systematic analysis of data or statistics.”
To make the best decisions in business and in life, we need to be adept at many different forms of thinking, including intuition, and we need to know how to incorporate many different types of inputs, including numerical data and statistics (analytics). Intuition and analytics don’t have to be seen as mutually exclusive at all. In fact, they can be viewed as complementary.
Let me give some examples of how intuition provides the spark for the analytical process.
Five Reasons Why Intuition Still Matters
- Knowing what to measure in the first place
- First, not even the NSA collects data on “everything”. I guarantee you a filter has been applied to narrow the set of inputs. God may have counted every hair on your head, but I seriously doubt anyone else has.
- Second, while data mining algorithms can discover notable patterns in huge data sets, only human intuition can discern between the useful patterns and the useless ones. They get their very “usefulness” from our goals and values.
- Knowing what the data is telling us (and what it’s not telling us)
Once we pick data to collect and metrics to analyze, what do the numbers tell us? We talked about preattentive attributes briefly – via data visualization, our intuition can be put to good use interpreting the important 1’s and 0’s in the databases we’ve meticulously built.
Using intuition in this way isn’t a perfect process though. Just as we might recoil from a garden hose that our instincts tell us is a snake, we can see signals in data that aren’t really there. Alternately, we can miss really important signals that are there. Just because intuition doesn’t work perfectly, though, doesn’t mean it should be discarded. We just need to hone our intuition for working with numbers, and we need to distrust it somewhat.
- Knowing where to look next
Jonas Salk, the American medical researcher who developed the first polio vaccine, had the following to say about intuition in his book Anatomy of Reality: Merging of Intuition and Reason:
Fig 3. From Jonas Salk’s Anatomy of Reality: Merging of Intuition and ReasonHe made a discovery that has saved the lives of countless people in the world, and he chalked up an important part of his success to intuition. Often the best outcome of an interaction with data is that we sense another, even better question to ask. And the process iterates. The realization of the next place to look can form in our mind like an intuitive spark. The light bulb analogy applies.
- Knowing when to stop looking and take action
For many types of questions or problems, we could continue to search for a solution ad nauseum. Think of a chess game. What’s the “best move” to make at a given point in the game? Russian chess Grandmaster Garry Kasparov knew something about this question, and here’s how he understood it, as stated in his book How Life Imitates Chess:
Fig 4. From Kasparov’s How Life Imitates ChessThere comes a point in time when it’s best to stop analyzing and make a move. Knowing when we’ve arrived at this point is a function of intuition. If we don’t have this intuitive switch, we can suffer from “analysis paralysis”, and then we go nowhere. We’ve all been there.
- Knowing what how to get our message across
A key part of the data discovery process is communicating our findings with others. We can use our intuition to choose the best message, channel, venue, visualization types, aesthetic elements, timing, tone, pace, etc. If we have a deep understanding of our audience, we will intuitively know what will get through to them, and what will fall on deaf ears. When we get it right, it can be a wonder to behold. Think Hans Rosling.
Crafting the communication is a creative process, and human intuition will need to be tapped to do it well.
Any process has an almost infinite number of variables that could be tracked and analyzed. On which should we spend our time? Knowing where to start can be a problem, especially if we know very little about the subject we’re dealing with.
One school of thought goes something like this: collect data on everything and let an algorithm tell you which to pay attention to.
Sorry, I don’t buy it.
Conclusion
For the reasons outlined above, I don’t believe that human intuition will ever be rendered obsolete. No matter how smart our algorithms get, no matter how sophisticated our tools or methods, the intuitive “spark” in the human mind will always be the key element in our thoughts, in our decisions and in our discoveries. Data and analytics can fuel these sparks, and they can provide a way to make sure we’re headed in the right direction, but they can’t replace human intuition. Not the way I understand it, anyway.
I don’t think the creators of the commercial would disagree with this point of view, so it likely comes down to semantics. Maybe the business owner in the commercial should have said: “We used to rely on intuition alone, now we combine it with analytics to make even better decisions.” Slightly less snappy, I know. But at least intuition doesn’t get thrown under the bus.
For a related example on chess and intuition, there was a study in the mid-90s by a pair of Carnegie Mellon researchers assessing Garry Kasparov’s play in simultaneous exhibitions relative to how he fared in standard tournaments. Chess uses Elo ratings for estimating player strength and win probability. Elo has proven accurate over the years, and there are variations used in World Cup Soccer (see eloratings.net) and college basketball (notably Jeff Sagarin’s system). In chess, an elite Elo rating is 2600+, master strength is 2200+, and a typical club-level player is around 1400.
In a simul, one person plays several games at once against a group of opponents. The exhibitor walks from board to board, playing moves sequentially until all the games are over. The exhibitor has to play fairly rapidly for this format to work, so there’s a stronger reliance on intuition and feel than in typical over-the-board chess. The opponents, on the other hand, get plenty of time to think and analyze, because if you’re one of 25 participants, you’ll have to wait each turn for the exhibitor to play 24 other moves before he comes back around to your game. In theory this should work in the opponents’ favor.
Most simuls involve a master taking on club-level opposition, so the rating gap between the exhibitor and the average Elo of the opponents may be 1000 points or more. Between 1985 and 1992, though, Kasparov played 9 simuls against fields with an average strength of 2320 or higher. These tended to be smaller fields of about 6 or 8 opponents, the strongest coming against the West German national team in 1992 (4 players with an average Elo of 2550). Kasparov’s rating was around 2750 at the time of the study, and his median performance rating in those 9 simuls was 2646 – lower than his standard rating but not by much. In his strongest performance, he notched a performance rating of 2786 against the Swiss national team – 5 wins and a draw against 6 players with an average Elo of 2394.
One interesting newer development in the post-Deep Blue computer chess world is that now there are freestyle tournaments where any combination of human and machine can enter. In these events the human-machine combinations tend to outperform computers alone, which is another example of how human intuition can complement raw computing power.
Fascinating stuff, thanks for commenting, Jay. Your point about freestyle chess and “advanced chess” is a great one, and explains my point exactly. In his book “How Life Imitates Chess”, Kasparov tells the story of the first such tournament where a team of two amateurs working with three computers were surprisingly victorious over much more skilled players. Intuition augmented by analytics is a powerful combination where the possibilities are vast and there is not much time to decide.