Defending Your Online Samples in Court
Having jumped back into the 1,016-page Reference Manual on Scientific Evidence from the Federal Judicial Center and the National Research Council (see last week’s article on Defending Your Statistics in Court), I keep finding fascinating nuggets of information worth sharing with our marketing and research colleagues of all stripes. Consider this snippet from the section on survey research:
A majority of the consumer surveys conducted for Lanham Act litigation present results from nonprobability convenience samples. They are admitted into evidence based on the argument that nonprobability sampling is used widely in marketing research and that “results of these studies are used by major American companies in making decisions of considerable consequence.”
What is the Lanham Act? It was passed in 1946 and prohibits false advertising and the misappropriation of trademarks. Not surprisingly, litigation related to this act relies on survey research to decide whether consumers are being deceived.
Even more interesting was a citation referenced in the snippet to an article entitled “Non-Probability Sampling Designs for Litigation Surveys” that was published 24 years ago in The Trademark Reporter. Remember, this is more than a decade before online surveys became ubiquitous:
… insofar as empirical research in the social and behavioral sciences is concerned, especially in those circumstances requiring that something be shown to the respondent, in an overwhelming proportion (i.e., 94 percent to 97 percent of the time), recognized scholarly and practitioner experts rely on non-probability sampling design. Accordingly, given the language and intent of Rule 703, triers-of-fact should have no problem admitting otherwise well-conducted non-probability surveys and according them due weight.
None of this is to suggest that we can do away with careful and rigorous approaches to sampling. But it is an excellent reminder: While representative samples are always essential, probability samples are not. Good research takes many forms, and if it is indeed good, it can withstand the toughest scrutiny in both boardrooms and courtrooms.
OTHER ARTICLES ON THIS TOPIC:
New York Times Shifts to Online Polling
