Thursday, November 19, 2009

Mavericks are best for crowd-sourcing

Maverick is the antonym of conformist or a culturally competent person. Synonyms include: individualist, nonconformist, free spirit, unorthodox person, original, eccentric; rebel, dissenter, dissident, enfant terrible; informal cowboy, loose cannon.

When we do a psychophysics experiment the old fashioned way in a lab, we want informants that are culturally competent persons. In fact, we are very careful in writing clear instructions, make sure the informants understand them, and check they follow the rules. The experimental conditions are strictly controlled so all informants perform exactly the same experiment.

When we do a psychophysics experiment the new way on the Web using crowd-sourcing, we get all beaten up by our colleagues and our papers keep getting rejected. "You are getting all those disruptive loose cannons out there, your results are meaningless." Well, we could almost reply "consider this formula:"

mean correlation of the aggregated responses to a world standard

I have to write "almost" because James Shilts Boster started writing his paper The Value of Cognitive Diversity: The Correlation of Local Aggregates with World Standards on May 6, 2004, but then as far as I know never got around to publish it.

The formula is for the mean correlation of the aggregated responses to a world standard. rxy is the average individual informant's correlation with the world standard, rxx is the average correlation among informants on the similarity judgment task, and N is the number of informants in the pool of aggregated responses.

This formula teaches that when N is small, like in the case of the old fashioned experiment, then we get the best correlation when all informants are culturally competent. Check!

However, when N is large, like in crowd-sourcing, then each new conformist informant does not contribute much to the correlation. Instead, it is the maverick informants, or better, the disagreement among informants that allows their aggregation to closely approximate the world standard. Surprise!

With this we call all rebels, dissenters, and mavericks out there and beg them to contribute to our color naming experiment at the link of their language on this page: http://www.hpl.hp.com/personal/Nathan_Moroney/mlcn.html