As they say, perhaps the future just isn’t what it used to be.

We can think about the futures (plural, because there is no one future) in terms of constants, cycles, trends and novelties.

These novelties are either (a) unanticipated events or (b) low likelihood, high impact events sometimes called “wild cards.”

I think Goldfarb is right in thinking that the next 10-30 years are full of novelty and wildcards.

But, if she’s right, then this means that straightline projections of past events and consumer behavior into the future could be very, very wrong. And, if this is the case, then market research as practiced is in very deep trouble.

Today at ARF Susan Whiting with Nielsen surfaced “privacy” as a core challenge to the industry. She’s right.

Given the recent stories regarding privacy and web “scraping”, I think we’re very close to the boiling point on this issue.

If doors are closed based on societal privacy concerns (eg some forms of listening), what new doors do we need to open?

Proprietary insights communities and prediction markets may be one set of answers that steps around some of these privacy concerns.

IBM’s purchase of SPSS (for $1.2 Billion!) was the initial spark to my thinking. After all, IBM is already heavily engaged in business consulting. And everyone has heard of the Smarter Planet concept. Most observers took the SPSS purchase as a bold move into predictive analytics, but I believe this was a more transformative purchase and a signal of where business consultancies are headed.

An even clearer signal is the formation of NM Incite, a joint venture between Nielsen and McKinsey. When Nielsen and McKinsey join forces, it is worth examining. Nielsen has the social media measurement prowess (BuzzMetrics) and McKinsey obviously has the business consulting brand.

A pattern is emerging.

Management-business consulting is clearly moving into the insights ecosystem in order to strengthen its core consulting offering by gaining proficiency with the analysis of new information flows (whether it is existing company data or open source social media data).

I cant think of a reason why this pattern won’t continue. The data inside and around companies continues to expand geometrically. There is strategic value in this data.

Insights ecosystem beware. Resistance (may) be futile. You (may) be assimilated.

Then again, the insights ecosystem may just take Ian Lewis’ good advice and move into a more consultative role themselves.

This is a noteworthy development.

The course will focus on social media, online communities and virtual research platforms.

Until now, the Burke Institute was fairly orthodox and conservative in its offerings. Their courses, all very solid, focused on surveys, focus groups and analytics – the traditional tools of the “Asking Epoch” of market research.

But, times change and most of market research is now behind advances of the wider insights industry. The market research industry may be struggling, but the wider insights industry is doing quite well and innovating rapidly.

The Burke Institute is a good proxy for tracking how traditional market research catches up.

I’m not criticizing them. They’re great folks that do great work. But they are playing catch up.

I anticipate that the Institute will roll out the following classes in the near future:

1. Mobile research tools and techniques
2. Insights Communities, UGC and Co-Creation (this deserves its own class)
3. Social media (again deserving its own class)
4. Neuromarketing or Biological Response Research
5. Eye Tracking in Ad, Packaging, Retail Environment and Menu Testing
6. Prediction Markets

Tracking the rapidity with which they roll out these new classes will be a good gauge of industry adoption.

In the meantime, market research professionals should support Burke’s evolution and attend the Sep. 30 – Oct. 1 class or send their junior staff. Send the market a signal. If the class oversubscribes dramatically, then the Burke Institute will react to the signal, offering more classes in this area andexpanding their new offerings.

Rubinson takes a look at the evolution of the ad agency and applies this evolutionary footpath to MR with media buying the analog to the data collection side of the business and creative the analog to the full service, analytics side of the business. It’s worth a read.

In my thinking about the future of the MR agency, I have developed at least five (5) plausible futures (not mutually exclusive):

1. Army of Davids: In this scenario, the larger firms keep consolidating and attempting to buy their future by acquiring hot new companies, but the systemic change and “free agent nation” overwhelms them. The goliaths are killed off by the Davids. The carriagemakers can’t adjust to the age of the automobile, and they are overhwelmed by new entrants, new technologies, new approaches (social media listening, etc.) and agile specialty firms.

2. Whuffie-Driven Free Agency: A second and more extreme “Army of Davids” scenario, Whuffie-driven free agency develops when individual researchers band together under a kind of digital guild in which each is rated by his-her clients. The resulting client score, their Whuffie (hat tip to Cory Doctorow’s “Down and Out in the MagicKingdom”) roughly determines an individual researcher’s compensation if and when they are chosen to do the work. Sound fantastical? Hardly. Consider: (a) eBay feedback profiles and (b) the mechanical turk. Now assume that a community of credentialed researchers (think PRC or similar) build an elance site on these principles. Insights rockstars, as opposed to large companies or even specialty firms, dominate the scene.

3. Convergence: Under the convergence scenario the MR industry is renamed, rebranded and redefined by a much larger collection of data-driven strategic consultants, including: management consulting, social media listening and analytics firms, predictive markets, game modelled consumer behavior research, foresight practices and IBM. Under this scenario, the first to combine all fields in an integrated way gains a significant advantage. The deepest pockets are the most favored here.

4. Incredible Shrinking MR: In this scenario the market research industry retains its current, distinctive, identity. It is not reinvented or transformed. Plenty of new innovations come along, but they’re not part of MR. MR labors on, like the dinosaurs after the first asteroid strike, oblivious of their doomed future.

5. DIY: Google, Facebook and future social media platforms enter the consumer research business by offering unimaginably massive panels of their users-members. With MR now a simple page on these social media sites, the era of DIY research is fully unleashed. Need a quick, global survey of fashion conscious, well-educated women? No problem. GoogleSurvey will collect data for you in 30 minutes after you have posted your (mandated) 5 questions or less survey. Need some collage-based qual done in 20 countries on a new tourist destination? No problem. 3 hours. Survey quality suffers, but the data is abundant. Insights? Well, those are a bit tougher to come by.

Tom Anderson promos the RIT a bit over at Next Gen Market Research. It’s worth a look.

The top two concerns are:

1. The non-representative nature of online samples
2. Quality of online sample is worse than most believe

Clearly these two concerns complement each other.

Although Jeffrey Henning is right about random samples vs. so-called “convenience samples”, the industry has clearly embraced “convenience samples” as the next best thing to purely random studies.

Stay tuned.

This year’s RIT study promises much more.

(Of course, I should disclose that my firm, StrategyOne, and I were pleased to be included in the survey design discussions for the RIT, and so I do have some “skin in the game.”)

That wouldn’t be my first word choice, but to get a sense of what leaders in the industry are thinking, you can check out their one word descriptors here.

You can also read Jeffrey Henning’s take at vovici here.

A quick summary:

transformational, rewind, qualitative, probability, listening, fulcrum, ethnographical, essential, consolidation

It is worth a read.

Macfarlane (Gfk), Poynter (Future Place) and Schillewaert (Insites Consulting) make the most interesting points in my opinion.

My word would be “REDEFINED”. The industry is not going through a gradual evolutionary shift, but something more akin to punctuated equilibrium. The tools, players, and industry definitions are all likely to change so much in the next five years that much of the industry will go by different names and be unrecognizable.

Upon further reflection, I might use the word “SCHUMPETER”, after the “Austrian School” economist that popularized the term “creative destruction” (to describe capitalism’s dynamism).

His colleague’s frustrations:

1. Too much email communication rather than face to face.
2. Increased speed of communication and deadline expectations.
3. Non-traditional competition from “non-researchers purporting to do or know how to do MR”.

The first is a general societal frustration.

I’ll address the 2nd and 3rd.

Speed:
Like it or not, expectations for more rapid turnaround time are growing. This is simply a function of the business environment and will only increase. But, I sense that there are somewhat different time expectations for work that is tactical (and in which the MR team is brought in late, usually for cover) and work that is of a more strategic nature. In my experience, the more strategic the work, the greater the understanding that time to contemplate is needed. Garbage in, garbage out. This is an advantage that the consulting community (McKinsey, BCG) have over traditional market research players. They may wind up using the same research tools, but they are brought in at a higher level and have the luxury of a greater time horizon for project completion.

I also strongly believe that one future of MR is the globally distributed research team. These teams will be able to leverage the 24 hour global clock in ways that could dramatically accelerate our delivery times. Right now most MR is regionally based with some offshoring of functions. The real change will come when the analytical and strategic teams are globally distributed.

Non Traditional Competition:
Get ready. Most of the non-traditional competitors will be in qualitative methods like communities, etc.

“Are the Kids Alright?”

What are they thinking about the future of market research?

Why aren’t we hearing from them?

My take:

I don’t think we’re asking their opinion and I’m not sure the industry is listening.

But, I would add another option somewhat related to “they’re too busy doing amazing stuff.”

“They’re in industries, businesses and functions that are not classified as ‘market research’ now, but will replace traditional ‘market research’ in a few years.”

Theoretical examples:

1. A MashUp whiz who is a baseball stats maven, has a boring job in accounting now, but will create a killer DIY research tool in 2012.

2. A gifted PHD candidate in anthropology that studies how women relate to food in their kitchens. She’ll go on in 2013 to build a stunning new “deep qual” firm.

3. A 25 year old Google math whiz that has never taken a survey course in her life, but will be tasked with architecting Google’s survey sampling methodology and DIY survey platform next year.

4. The 27 year old Boston Consulting Group wunderkind that is a virtuoso at synthesizing multiple data sets in his consulting gigs, but would never identify himself as a “market researcher” or in “market research”.

5. The 26 year old online gaming programmer that loves building simple simulation games for his buddies, but in 2013 will have the hottest new research interface on the planet. His company (Simusurveys) won’t ask respondent questions at all. Instead, it will have them play fun simulations. In fact, many people will play these surveys willingly as entertainment knowing that they are actually part of niche research efforts.

6. A 22 year old self-described “fashionista” in Miami that will create a global, invitation only, communnity for only the most cutting edge hipsters. When a high fashion corporation realizes she is sitting on a gold mine, they’ll buy her community for top dollar and hire her to maintain it. She will become their lead trends expert among the trendiest trendsetters. Overnight she’ll be a market research executive, but she’ll never use the term.

I could go on, but I think this is what part of the transformation will look like.

Highlights:

1. Only 35% of executives describe their in-house MR function as “best in class.” (see page 4)

2. Although corporations should strive to move their MR departments from “order takers” to a “source of competitive advantage”, “almost 90% of the companies…follow a more traditional approach to market research” (Translation: 90% are stuck in “order taker” mode.) (see page 4)

3. “On average, only 20 to 35 percent of a company’s market-research budget is devoted to strategic studies.” (see page 5)

4. There is a CLEAR need for what I have described as “Insight Management” – getting the most out of past research and mining it for synthesized insights. From the report: “Money is spent on research reports that languish on dusty shelves because the data rarely yield actionable plans. Many companies accumulate such a quantity of research and data that quality is inevitably elusive, each study providing a limited, tactical perspective on the consumer, with little integration of synthesis.” (see page 6)

5. Answering the “So What?” Question. While 73% of MR staff said they consistently answer the question “so what?” about the data they provide, only 34% of the business staff thought they closed the loop and answered this question. (see page 14)

6. Tactical, Not Strategic. Unfortunately, 59% of market researchers agree with the statement “we spend the majority of consumer insight time and effort on decisions with near-term impact.” This means that MR departments are completing numerous small studies and not given the time to explore the bigger picture. This also creates a low status cul-de-sac for MR departments as it is difficult to have solid strategic input when tactical, short term studies are the focus. (see #3 above)

7. The Rise of Polymaths. Interestingly, the report suggests that a desired “strategic foresight organization” (the final, 4th phase of an MR department’s evolution) will employ a more diverse range of researchers from statistics, anthropology, sociology, marketing etc.

In its entirety the BCG report paints a fairly stark picture of the current situation. It is true that market research needs to evolve, but corporate cultures will need to support this evolution as well.