Culling the herd: using real-world randomized experiments to measure social bias with known costly goods

Peer ratings have become increasingly important sources of product information, particularly in markets for information goods. However, in spite of the increasing prevalence of this information, there are relatively few academic studies that analyze the impact of peer ratings on consumers transacting in real-world marketplaces. In this paper, we partner with a major telecommunications company to analyze the impact of peer ratings in a real-world video-on-demand market where consumer participation is organic and where movies are costly and well known to consumers. After experimentally changing the initial conditions of product information displayed to consumers, we find that, consistent with the prior literature, peer ratings influence consumer behavior independently from underlying product quality. However, we also find that, in contrast to the prior literature, there is little evidence of long-term bias as a result of herding effects, at least in our setting. Specifically, when movies are artificially promoted or demoted in peer rating lists, subsequent reviews cause them to return to their true quality position relatively quickly. One explanation for this difference is that consumers in our empirical setting likely had more outside information about the true quality of the products they were evaluating than did consumers in the studies reported in prior literature. Although tentative, this explanation suggests that in real-world marketplaces where consumers have sufficient access to outside information about true product quality, peer ratings may be more robust to herding effects and thus provide more reliable signals of true product quality than previously thought.