How Do You Really Feel?

While on the subject of Stanley Milgram, there is another very interesting experimental technique he pioneered which I think is directly relevant to product managers and marketers. The issue is: how can you determine how people truly feel about something? Asking them (e.g. via surveys or focus groups) can be problematic and all sorts of biases may be introduced. For instance, how you frame the question has a strong impact on the results (nice little example of framing bias by Paul Kedrosky here).

While market analysts can adjust for these biases in some cases, things become particularly troublesome when the questions are more personal and/or sensitive. People tend to respond in accordance with a social desireability bias, which basically means they will tell you what they believe to be a more socially acceptable response, rather than what they really think. How do you estimate the amount of popular support for white supremacy or neo-Nazi groups? Almost no one who holds such a view will own up to it. Election polls in many countries severely understate the support for extremist political parties for the same reason (including the famous 2002 French election in which the extreme right winger Jean-Marie Le Pen came second in the preliminary round of voting - a shocking result. In the United States, a rough analogy would be if David Duke won the Republican primary).

Milgram devised a particularly clever experimental technique that attempted to deal with these types of issues. It was called the "lost-letter" technique, and it worked as follows. Milgram would address letters to a fictitious group whose affiliation would be clear from its name (e.g. "Society For The Advancement of White Supremacy"). The letters would be stamped and addressed to a Post Office box that Milgram set up ahead of time. Hundreds of these letters would then be placed at selected locations, looking like they had somehow got lost and just needed to be placed in a mailbox to be sent along its way. He then monitored the Post Office Box to see how many letters came in.

Milgram guessed that a degree of sympathy for the organization mentioned on the envelope would make it more likely that someone would actually send the letter on its way rather than ignore it. And the anonymous and indirect nature of the transaction would make it a lot easier to act on your true feelings about an issue, not just a socially accepted feeling. To factor out the "noise" from non-responses and other random events, Milgram would do the same thing with another set of letters, this time addressed to a completely neutral sounding organization (e.g. "Industrial Corp, Inc."). This would be the control data against which the responses from the sensitive letter could be measured. The greater the response, the more the support for a particular viewpoint.

The responses to the various sets of letters he tested with were not particularly revelatory. It's the technique that was his real accomplishment, not the results of the experiment.