Biases That Change How We Read Data

Data doesn't usually mislead us because it's wrong. It misleads us because we interpret it through hidden biases.

In this series, we'll explore some of the most important statistical biases that affect decision-making across business, development, and public policy, and why recognizing them leads to better evidence and better decisions.

1. Selection Bias: Are You Only Hearing from the People Who Showed Up?

Imagine evaluating a training program by interviewing only those who attended the final session. The results may look overwhelmingly positive—but what about those who dropped out?

Selection bias occurs when the people included in an analysis are not representative of the entire population.

At Insights, we pay close attention to who is not represented in the data, because understanding missing voices is often just as important as analyzing the responses we receive.

2. Nonresponse Bias: Silence Can Change the Story

Suppose 90% of respondents say they are satisfied with a service. That sounds impressive.

But what if 35% of the selected respondents could never be reached? If those households systematically differ from those who responded, the findings may no longer represent the target population.

At Insights, response rates and respondent characteristics are not just quality-control metrics—they are essential for interpreting findings responsibly.

3. Survivorship Bias: Learning Only from Success

Imagine evaluating businesses that received grants five years ago—but only those that are still operating.

The businesses that failed are no longer part of the analysis.

The result? The program may appear far more successful than it actually was.

At Insights, we believe strong evaluations should seek evidence from both successes and failures. Often, the most valuable lessons come from understanding why an intervention did not achieve its intended results.

4. Margin of Error: Every Estimate Has a Range

A survey reports that 64% of beneficiaries are satisfied.

Is the true value exactly 64%?

Probably not.

Every survey estimate contains uncertainty. Understanding that uncertainty is just as important as understanding the estimate itself.

At Insights, we emphasize interpreting survey findings within their statistical context rather than treating every percentage as an exact measurement.

5. Correlation Is Not Causation

Two indicators may move together, but that does not mean one caused the other.

For example, communities with higher incomes may also report greater satisfaction with public services. That does not automatically mean income caused the improvement.

At Insights, we distinguish between observed relationships and evidence of causal impact, ensuring that conclusions remain grounded in the study design.

6. Regression to the Mean: Not Every Improvement Is the Result of an Intervention

Projects often focus on the poorest-performing schools, facilities, or communities.

When those locations improve later, it is tempting to credit the intervention entirely.

Sometimes that is true. Sometimes part of the improvement reflects natural variation over time.

At Insights, we interpret change carefully, recognizing that meaningful evaluation requires understanding both intervention effects and underlying statistical patterns.

7. Base Rates: Context Matters

Imagine a screening tool that correctly identifies a rare condition 99% of the time.

That sounds excellent.

But if the condition itself is extremely uncommon, many of the positive results may still turn out to be false alarms.

Strong decision-making requires understanding not only how accurate a tool is, but also how common the event is to begin with.

At Insights, we believe that evidence gains meaning only when interpreted within its broader context.

8. Confounding: When Something Else Is Driving the Result

Suppose one community achieves much better outcomes than another.

Was it because of the project?

Or because that community already had stronger infrastructure, higher income, or better access to services?

Confounding occurs when another factor influences both the intervention and the outcome.

At Insights, identifying and accounting for these factors is a fundamental part of producing credible analysis rather than simply reporting observed differences.