The reasons because audience tests do not work

Any time you see “test audiences revealed that…” there’s more than a 95% of chances it’s going to be useless – here’s the main and the two secondary reasons. And then, obviously, also a solution.

The main reason: Hawthorne effect

The Hawthorne effect is a voluntary psychological bias (people are aware and willing to do it) that kicks in whenever a subject is aware of being tested.

The two possible outcomes are:

1. The subject, afraid to look bad during the test, will modify his answers to align with trendy opinions rather than what he genuinely thinks (conformity). Important: people, outside exceptions, are terrified of looking bad. Especially if strangers, and of a somewhat high status (e.g. Advertising professionals).
2. The subject, knowing that his test is going to influence someone else, will alter his answers to skew the final judgement towards his favorite conclusion – rather than saying what he genuinely thinks about the topic.

The 1st secondary reason: tunnel vision

The more we think about something, the more we get acclimatized to what we’re analyzing – including errors and biases. Or, as they call it: job conditioning. This means that: there’s a very limited window within which answers are going to be reliable.

This is the very same mechanism for which, many times, the work we’ve done the day before doesn’t look that great anymore the day after: the day before, we might have worked so much that we stopped noticing errors. The day after, instead, after being well rested, everything popped up. Within the music production industry, is called “ear fatigue”: the more you mix something, the less you notice errors – and so, you have to be very careful about being quick and taking rests.

Solutions

Given those problems, here’s 2 quick workarounds and a not so quick one:

Quick ones

Create test in which the audience is not aware they’re being tested

Do not create tests who require an excessive amount of effort

Here’s a quick example: want an honest opinion about a trailer? Air your trailer in a TV shop, and observe carefully how the clientele reacts to it. And, afterwards, have your analysts blend into the clientele and ask about the trailer in a dismissive way (e.g. Hey that’s a great TV! What do you think about it? I wanted one for my boy, as graduation gift… [gets answer] Oh, by the way: what the heck was that trailer?!?!?)

Not so quick one

And not only “not so quick”: it’s also going to be very 1984ish.

Outside exceptions, the emotional processing unit of our brain is largely outside our control – in detail: the limbic system (“reptilian brain”) interprets emotional responses to what the neocortex understood (gray matter). Which means that, unless the test audience is ex KGB agents or Oscar winning performers, their immediate emotional physiological reaction during the screening is very likely to be genuine. Hence: hook them up to a polygraph and an EEG, and evaluate what data says.

Here’s a practical example:

A hates B > A is shown X, a great work done by B > A is amazed by X’s quality, and his physiological parameters are altered by that (e.g. Heart-beat goes up, “goose bumps”… You know the drill) > A remembers that X has been done by B, focuses, and ignores his feelings so to normalize the cognitive dissonance* > When A is asked what he thinks about X, he’s going to answer “It sucks!“, because he doesn’t want to miss an opportunity to get back at B. Even though, in truth, X was great.

*cognitive dissonance is when we’re harmed by having conflicting thoughts (e.g. “I’d like to be A, but I’m B instead”). It is solved by either taking action (assertiveness/competence), or pulling out excuses with which sweep the problem under the rug (weakness/incompetence).

…solutions?

These solutions have a “problem”: tests made like so, are likely to give truthful outcomes. And …do people really want them? Many times, tests have huge conflicts of interests: they’re purposely skewed so to find the conclusion the testers wanted. Reliable tests, instead, might say the exact opposite of what were you thinking – because that’s how science works: scientific research is discovery. And you might discover things you haven’t imagined – or liked. It’s a bit like reading a book: you might have an idea of what’s going to happen in the next pages, but you’ll never know until you read – and you might not like what you’re going to find out.

And this, by the way, is the third problem: many times, testers are biased and not willing to be honest about using and reporting test audiences.