Defending Your Statistics in Court
There is nothing worse than presenting research findings to an audience predisposed to hate it. Maybe you’ve had that experience of being in a boardroom of hostile managers? They don’t like what you’re about to say, so they pick at every methodological decision, shifting focus away from the story that the data have to tell. It is one reason I left the world of academia not so long ago.
But what if you were hired to play that very game, and you’ve got the expertise to win? It can be exhilarating, which is why research to support litigation can be satisfying work. “We know this stuff inside and out,” I tell our team. “Just be ready, with the authority we have, to speak to every detail of the research.”
Those details are nicely summarized in the Reference Manual on Scientific Evidence, compiled by the Federal Judicial Center and the National Research Council. Several months back we described the section on Survey Research, which focuses on survey design and sampling. Here we offer an overview of the 91-page Statistics section.
If nothing else, this outline (shown verbatim from the section’s Table of Contents) offers a nice primer on everything one needs to know about statistics to do credible quantitative market research:
I. Introduction
A. Admissibility and Weight of Statistical Studies
B. Varieties and Limits of Statistical Expertise
C. Procedures That Enhance Statistical Testimony
1. Maintaining professional autonomy
2. Disclosing other analyses
3. Disclosing data and analytical methods before trial
II. How Have the Data Been Collected?
A. Is the Study Designed to Investigate Causation?
1. Types of studies
2. Randomized controlled experiments
3. Observational studies
4. Can the results be generalized?
B. Descriptive Surveys and Censuses
1. What method is used to select the units?
2. Of the units selected, which are measured?
C. Individual Measurements
1. Is the measurement process reliable?
2. Is the measurement process valid?
3. Are the measurements recorded correctly?
D. What Is Random?
III. How Have the Data Been Presented?
A. Are Rates or Percentages Properly Interpreted?
1. Have appropriate benchmarks been provided?
2. Have the data collection procedures changed?
3. Are the categories appropriate?
4. How big is the base of a percentage?
5. What comparisons are made?
B. Is an Appropriate Measure of Association Used?
C. Does a Graph Portray Data Fairly?
1. How are trends displayed?
2. How are distributions displayed?
D. Is an Appropriate Measure Used for the Center of a Distribution?
E. Is an Appropriate Measure of Variability Used?
IV. What Inferences Can Be Drawn from the Data?
A. Estimation
1. What estimator should be used?
2. What is the standard error? The confidence interval?
3. How big should the sample be?
4. What are the technical difficulties?
B. Significance Levels and Hypothesis Tests
1. What is the p-value?
2. Is a difference statistically significant?
3. Tests or interval estimates?
4. Is the sample statistically significant?
C. Evaluating Hypothesis Tests
1. What is the power of the test?
2. What about small samples?
3. One tail or two?
4. How many tests have been done?
5. What are the rival hypotheses?
D. Posterior Probabilities
V. Correlation and Regression,
A. Scatter Diagrams
B. Correlation Coefficients
1. Is the association linear?
2. Do outliers influence the correlation coefficient?
3. Does a confounding variable influence the coefficient?
C. Regression Lines
1. What are the slope and intercept?
2. What is the unit of analysis?
D. Statistical Models
