2.4 From Simulation to Theory

In the randomization tests and bootstrap confidence intervals chapters, we used simulation to answer two fundamental questions of inference: Is there an effect? (randomization tests) and How big is the parameter? (bootstrap confidence intervals). Both methods relied on the computer to generate distributions — randomization distributions and bootstrap distributions — that helped us quantify uncertainty. In this chapter, we step back to see the bigger picture. We examine the shape, center, and spread of these simulated distributions and discover a remarkable pattern: under certain conditions, they all look approximately normal (bell-shaped). This observation, formalized as the Central Limit Theorem, is the bridge from the simulation-based methods of Unit 2 to the formula-based methods of Unit 3.

Key Concepts

Explain the conclusion of the Central Limit Theorem (CLT)
Recognize when the CLT can be applied to a study
Identify when each of the three inference procedures should be used

The Central Limit Theorem

The Central Limit Theorem (CLT) stats that when we take sufficiently large random samples from a population, the sampling distribution of many common statistics (such as $\widehat{p}$ or $\overline{x}$) is approximately normal (bell-shaped), regarless of the shape of the underlying population distribution

The Central Limit Theorem (CLT) tells us that for any population distribution (no matter how skewed or strange the population is), if we repeatedly take new random samples from this distribution and calculate the mean or proportion each time, then:

As sample size increases, the sample average or proportion should get closer to the population average.
As sample size increases, the averages or proportions will be less spread out.
As sample size increases, the distribution of the sample averages or proportions will look more like a bell-shaped (normal) curve.

Conditions for the CLT

The bell-shaped approximation does not apply in all situations. Two conditions must hold:

Independent observations. The observations in the sample must be independent of each other. This is typically guaranteed by random sampling from a population, or by random assignment in an experiment.

Large enough sample. The sample size cannot be too small. What counts as “large enough” depends on the context:
- For proportions: we generally need at least 10 expected successes and 10 expected failures.
- For means: the rule of thumb depends on how skewed the populations is. We will generally use $n>30$ for STAT 145.

Class Example 2.4.1: Visualizing the CLT

Go to the Sampling Distribution Lab in StatLens. We will use the Right-Skew population shape. We wish to visualize the distribution of the sample mean.

Set the sample size to $n=5$ and draw 1000 samples. What is the shape of the sampling distribution?

Set the sample size to $n=30$ and draw 1000 samples. How does the shape of the sampling distribution change?

Set the sample size to $n=100$ and draw 1000 samples. How does the shape of the sampling distribution change?

Which sample size would you use if you wanted the shape of the sampling distribution to be bell-shaped

Now, selected a Normal population shape. Set the sample size to $n=5$ and draw 1000 samples. How does the shape of this sampling distribution compare to the sampling distribution shape from part (a)?

Set the sample size to $n=30$ and draw 1000 samples. How does the shape of this sampling distribution change from part (e). How does it compare to the sampling distribution shape from part (b)?

Comparing randomization, bootstrap, and mathematical approaches

We now have three tools for statistical inference, each grounded in the same core ideas but using different mechanisms:

Comparison of inference methods
	Randomization	Bootstrap	Mathematical model
Question answered	Is there an effect? (hypothesis test)	How big is the parameter? (confidence interval)	Both
How variability is generated	Shuffle labels to simulate $H_0$	Resample with replacement from sample	Use formulas based on the normal distribution
What it models	Variability due to random assignment	Variability due to random sampling	Variability predicted by mathematical theory
Center of distribution	Null hypothesis value	Sample statistic	Depends on context (null values for tests, point estimates for CIs)
Conditions required	Minimal	Minimal	Independence + large sample
Computational cost	Moderate (need many simulations)	Moderate (need many resamples)	Low (just plug into formula)

All three methods answer the same fundamental question: How much does the statistic vary? They just measure that variability in different ways.

Randomization asks: “How much would the statistic vary if the null hypothesis were true?”
Bootstrap asks: “How much would the statistic vary across different samples from this population?”
Mathematical models ask: “How much does theory predict the statistic would vary?”

When conditions are met, all three give similar answers. The mathematical approach is faster and requires no simulation, which is why it is widely used. But the simulation approaches are more flexible and work even when conditions for the mathematical model are not met.

When is the normal approximation good enough?

The mathematical approach to inference (which we will develop in Unit 3) replaces simulation with the normal distribution. Instead of generating thousands of shuffles or resamples, we use a formula to compute the standard error and then rely on the bell curve to find p-values or construct confidence intervals.

But when can we trust this shortcut?

The success-failure condition for proportions

For a sample proportion $\hat{p}$, the sampling distribution is approximately normal when:

\[\text{Expected Successes: } np \geq 10 \quad \text{and} \quad \text{Expected Failures: }n(1 - p) \geq 10\]

where $n$ is the sample size and $p$ is the proportion of interest. In practice, since $p$ is unknown, we use $\hat{p}$ as a stand-in:

\[n\widehat{p} \geq 10 \quad \text{and} \quad n(1 - \widehat{p}) \geq 10\]

This is called the success-failure condition.

The sample size condition for means

For a sample mean $\overline{x}$, the CLT tells us the sampling distribution is approximately normal when:

The sample size is “large enough,” and
The observations are independent.

A common guideline is $n \geq 30$, but this is only a rough rule of thumb. If the population distribution is nearly symmetric, the normal approximation can work well even for smaller samples. If the population is highly skewed or has extreme outliers, a larger sample may be needed.

Population shape	Minimum $n$ for normal approximation
Symmetric, no outliers	$n \geq 15$ or even smaller
Slightly skewed	$n \geq 30$
Highly skewed or heavy-tailed	$n \geq 60$ or more

When in doubt, use simulation (bootstrap) to check whether the normal approximation is reasonable.

Class Example 2.4.2: Applying the CLT

For each of the following indicate whether the CLT can be applied. Why or why not?

We are interested in approximating the sampling distribution of the sample mean with samples of size 10.

We are interested in approximating the sampling distribution of the sample mean with samples of size 100.

We are interested in approximating the sampling distribution of the sample proportion when $n = 62$ and $\widehat{p} = 0.05$.

We are interested in approximating the sampling distribution of the sample proportion when $n = 1000$ and $\widehat{p} = 0.5$.

We are interested in approximating the sampling distribution of the sample minimum with samples of size 100.

Summary

Randomization distributions, bootstrap distributions, and theoretical sampling distributions all tend to be approximately bell-shaped (normal) under common conditions. This is the Central Limit Theorem (CLT).
The CLT states that for sufficiently large samples with independent observations, the sampling distribution of many common statistics is approximately normal, regardless of the shape of the population.
The success-failure condition ($n\hat{p} \geq 10$ and $n(1-\hat{p}) \geq 10$) is the practical check for whether the normal approximation is appropriate for proportions.
The three inference approaches — randomization, bootstrap, and mathematical models — answer the same fundamental questions. When conditions are met, they produce similar results. Mathematical models are faster; simulation methods are more flexible.
This chapter bridges Unit 2 (simulation-based inference) and Unit 3 (formula-based inference). The CLT is the theoretical justification for the formulas we will use in Unit 3.