Most companies waste hundreds of thousands of dollars and man hours on unnecessary or ill-conceived market research.
I know whereof I speak because for six years, I worked in a marketing group inside the software division of a huge high tech firm. Our group spent around $2 million a year on market research in today’s dollars: a combination of custom focus groups, brand searches, and surveys, along with subscriptions to analyst newsletters and special reports.
Eventually, I ended up with control over that market research spending, which meant that I decided what we should be researching, who we would hire, and how the data would be presented to decision-makers. I took the assignment because I believed that a better understanding of our market would help the company create better products and services.
What I quickly discovered, however, is that top management did not really want a better understanding of the market. What they wanted was ammunition that would retrospectively support the wisdom of decisions they had made in the past or reinforce positions they’d taken, or planned to take, in turf battles over budget and organizational structure.
Confirmation bias, of course, is present in all decision-making, but this was more than that. Rather than just sorting data so that it gelled with preconceptions, management wanted us to actively find and distort data so that it reinforced those preconceptions, but in such a way that it all looked “objective.”
For example, my predecessor hired what she called a “boutique” market research firm (i.e. a guy who’d just hung his shingle) to prove that our company made more money in “digital publishing” than Apple–an utterly crazy notion, since our company made almost all of its money selling minicomputers.
The solution they came up with was to count as a “publishing system” any sale that included a postscript printer. Thus, if a customer spent $1 million on a mainframe and included, in the order, a $4,000 printer, we counted that entire $1 million as a “publishing system.” As a result, our group asked for, and received, several million dollars to market “publishing systems.”
Just to be clear: in the real world we probably sold less than $100,000 in systems actually used for publishing; Apple was already a multibillion dollar company. And yet, this crazy idea “flew” because our marketing group manipulated market data in a way that seemed plausible to top management–who loved the idea that we were “winning” against Apple.
As soon as I controlled of the research budget, I immediately fired the “boutique” consultants and hired real-world research firms that could actually give us data that reflected the real world. (Among those companies I hired was Delphi Research, headed at the time by my fellow Inc.com columnist, Tom Koulopoulos.)
Unfortunately, it proved impossible (at least for me) to wean my company’s top management away from “confirmation bias” market research. The reason was simple and very human: accepting data that ran contrary to previous assumptions meant that management would have to admit the CEO’s “vision” for the company (e.g. “let’s beat Apple”) was, and always had been, unrealistic.
And that wasn’t going to fly. As the resident market research expert, I was once invited to a super-important, VP-level strategy session to figure out how to pull the company out of impending bankruptcy. I kid you not, the meeting started with a statement “the purpose of this meeting is to determine the best way to tell the CEO what we know he wants to hear.”
I tried every way possible–charts, documents, props, animations, etc.– to get the decision-makers to look at the real world data and then make recommendations based upon that data that might pull the company out of its self-induced tailspin.
Finally, in complete desperation, I said, and I quote myself: “The fact that top management is sometimes ignorant or intransigent does not obviate us from the responsibility to educate them when necessary.”
I have no idea why I thought that might work (they just stared at me…) but I was at my wits end. In any case, the company went bankrupt not long afterward. Maybe if I had done a better job at injecting reality into discussions of the company’s future strategy, they’d still be around. But it was beyond my capability to wake these guys up.
Now, make no mistake about it. Everyone in that company–especially top management–were the “smartest guys in the room.” The CEO, in fact, was eulogized in the title of a bestselling book as “The Ultimate Entrepreneur.” The situation was just an exaggerated example of what happens every day inside organizations large and small.
In short, market research makes your company dumber, rather than smarter, if:
- Your goal is to confirm a decision you’ve already made
- The data is intended primarily as ammunition for a turf war.
- You’re unwilling to make potentially painful changes to your organizations, products, services, processes, and business model.
If you approach market research with true objectivity, however, it can help you understand what’s going on in the world and suggest how you can take advantage of it. You’ve just got to check your confirmation bias before getting the process started.