8  Decision Errors

In the previous chapter we built the logic of a hypothesis test: state a null and an alternative, measure the evidence with a p-value, and reject \(H_0\) when the p-value falls below the discernibility level \(\alpha\). But a decision based on data can be wrong in two different ways. This chapter names those two mistakes — the Type I error and the Type II error — organizes them in a decision table, gives each one an error rate (\(\alpha\) and \(\beta\)), and examines the tradeoff between them. Naming these errors is what lets us choose a discernibility level deliberately rather than out of habit.

StatLens: Decision Errors simulator. Open the Decision Errors simulator (or the open sandbox). It uses a plain one-proportion framing — \(H_0\): \(p = 0.50\) vs \(H_A\): \(p > 0.50\) — and lets you decide whether the alternative is actually true (and at what success rate), pick \(n\) and \(\alpha\), then run many studies. Each study is one dot: green if the test detected a real effect, gray if it missed (Type II), red if it false-alarmed under a true \(H_0\) (Type I). The hit/miss/false-alarm strip is the picture this chapter is about. (Power gets a fuller treatment in Chapters 29–31; the analytical Power & Error Visualizer is there if you want a quick preview.)

8.1 Type I error

Recall the structure of a hypothesis test. We weigh two competing claims:

• \(H_0\) (the null hypothesis): there is no effect, no difference, or no relationship.
• \(H_A\) (the alternative hypothesis): there is an effect, difference, or relationship.

We collect data, compute a test statistic, and find a p-value — the probability of observing results at least as extreme as ours if \(H_0\) were true. If the p-value is below our chosen discernibility level \(\alpha\), we reject \(H_0\).

But rejecting \(H_0\) when it is actually true is a mistake, and that mistake has a name.

A Type I error occurs when we reject the null hypothesis \(H_0\) even though \(H_0\) is actually true. In other words, we conclude there is an effect when in reality there is none.

The probability of a Type I error is controlled by the discernibility level \(\alpha\). When we set \(\alpha = 0.05\), we are accepting a 5% chance of making this kind of mistake — declaring a “discovery” that is really just noise. In the sex discrimination study of the previous chapter, a Type I error would mean concluding that the managers’ decisions depended on the candidate’s sex when, in truth, they did not.

8.2 Type II error: missing a real effect

Suppose a new medication truly does lower blood pressure compared to a placebo. A clinical trial is conducted, data are collected, and the researchers perform a hypothesis test. But the p-value comes back at 0.14 — not small enough to reject \(H_0\) at the \(\alpha = 0.05\) level. The researchers conclude: “We did not find sufficient evidence that the medication works.”

But the medication does work. The study simply failed to detect the real effect. This is a Type II error.

A Type II error occurs when we fail to reject the null hypothesis \(H_0\) even though the alternative hypothesis \(H_A\) is actually true. In other words, there is a real effect, but our test misses it.

Type II errors are sometimes called “false negatives” — the test comes back negative (fail to reject \(H_0\)) when the truth is positive (the effect exists). In medical screening, this is analogous to a test that says “no disease” when the patient actually has the disease. In a court system, it is analogous to acquitting a guilty person.

8.3 The decision table

Every hypothesis test ends with one of two decisions: reject \(H_0\) or fail to reject \(H_0\). And the truth is one of two realities: \(H_0\) is true or \(H_A\) is true. Combining these gives four scenarios:

Table 8.1: Four possible outcomes of a hypothesis test.

	\(H_0\) is true	\(H_A\) is true
Reject \(H_0\)	Type I error	Correct decision
Fail to reject \(H_0\)	Correct decision	Type II error

Two of the four outcomes are correct decisions:

• Rejecting \(H_0\) when \(H_A\) is true — we correctly detected the real effect.
• Failing to reject \(H_0\) when \(H_0\) is true — we correctly avoided a false alarm.

The other two outcomes are errors:

• Type I error (upper-left): We rejected \(H_0\) but it was actually true. False alarm.
• Type II error (lower-right): We failed to reject \(H_0\) but \(H_A\) was actually true. Missed detection.

In a US court, the defendant is either innocent (\(H_0\)) or guilty (\(H_A\)). What does a Type I error represent in this context? What does a Type II error represent? Table 8.1 may be useful.

---

If the court makes a Type I error, this means the defendant is innocent (\(H_0\) true) but wrongly convicted. A Type II error means the court failed to reject \(H_0\) (i.e., failed to convict the person) when they were in fact guilty (\(H_A\) true).

A pharmaceutical company tests whether a new drug reduces cholesterol more than a placebo. They set up \(H_0\): the drug has no effect, and \(H_A\): the drug reduces cholesterol. Describe what a Type I error and a Type II error would mean in this context.

Show answer

A Type I error would mean concluding the drug reduces cholesterol when it actually does not — the company might invest millions in producing an ineffective drug. A Type II error would mean failing to detect a real cholesterol-lowering effect — the company might shelve a drug that actually works.

A quality control inspector tests whether a batch of light bulbs has a defect rate above 2%. She sets up \(H_0\): defect rate \(\leq 0.02\) and \(H_A\): defect rate \(> 0.02\). Which error is worse in this context: Type I or Type II? Why?

Show answer

A Type II error is arguably worse here: the inspector would fail to detect that the defect rate is too high, and defective bulbs would be shipped to customers. A Type I error means rejecting a good batch, which wastes product but doesn’t harm customers. The relative costs depend on the specific situation, but in quality control, missing a real problem (Type II) often has more serious consequences.

8.4 Error rates: \(\alpha\) and \(\beta\)

We give the probabilities of the two error types their own Greek-letter names:

The Type I error rate, denoted \(\alpha\), is the probability of rejecting \(H_0\) when \(H_0\) is true:

\[\alpha = P(\text{reject } H_0 \text{ when } H_0 \text{ is true})\]

The Type II error rate, denoted \(\beta\), is the probability of failing to reject \(H_0\) when \(H_A\) is true:

\[\beta = P(\text{fail to reject } H_0 \text{ when } H_A \text{ is true})\]

The discernibility level \(\alpha\) is something we choose before the test — it is under our direct control. Common choices are \(\alpha = 0.05\), \(\alpha = 0.01\), and \(\alpha = 0.10\).

The Type II error rate \(\beta\), on the other hand, is not something we directly choose. It depends on several factors that we will explore shortly: the true effect size, the sample size, the variability in the data, and the chosen discernibility level \(\alpha\). Unlike \(\alpha\), \(\beta\) is usually unknown because we don’t know the true state of the world — if we knew \(H_A\) were true and knew the exact effect size, we wouldn’t need to run the test.

This asymmetry between \(\alpha\) and \(\beta\) is a fundamental feature of the hypothesis testing framework. We control one error rate directly (\(\alpha\)) and try to manage the other (\(\beta\)) through careful study design.

8.5 The tradeoff between \(\alpha\) and \(\beta\)

How could we reduce the Type I error rate in US courts? What influence would this have on the Type II error rate?

---

To lower the Type I error rate, we might raise our standard for conviction from “beyond a reasonable doubt” to “beyond a conceivable doubt” so fewer people would be wrongly convicted. However, this would also make it more difficult to convict the people who are actually guilty, so we would make more Type II errors.

How could we reduce the Type II error rate in US courts? What influence would this have on the Type I error rate?

Show answer

To lower the Type II error rate, we want to convict more guilty people. We could lower the standards for conviction from “beyond a reasonable doubt” to “beyond a little doubt”. Lowering the bar for guilt will also result in more wrongful convictions, raising the Type I error rate.

The example and guided practice above provide an important lesson: if we reduce how often we make one type of error, we generally make more of the other type.

This tradeoff is not just a courtroom analogy — it is baked into the mathematics of hypothesis testing. Here is the core intuition:

• Lowering \(\alpha\) means we require stronger evidence (a smaller p-value) before rejecting \(H_0\). This makes us less likely to reject \(H_0\) when it is true (good), but also less likely to reject \(H_0\) when it is false (bad). So \(\beta\) increases.
• Raising \(\alpha\) means we accept weaker evidence for rejecting \(H_0\). This makes us more likely to detect real effects (\(\beta\) decreases), but also more likely to make false discoveries (\(\alpha\) increases).

The \(\alpha\)-\(\beta\) tradeoff. For a fixed sample size and effect size, decreasing \(\alpha\) increases \(\beta\), and vice versa. You cannot reduce both error rates simultaneously without changing something else — typically the sample size.

To see why this makes sense, imagine a number line representing all possible values of a test statistic. Under \(H_0\), the test statistic is centered at some null value. The rejection region is the set of values far enough from the null to trigger rejection. Making \(\alpha\) smaller means shrinking the rejection region — fewer values lead to rejection. But if \(H_A\) is true, the test statistic is centered somewhere else. A smaller rejection region means less of the alternative distribution falls inside it, so we are less likely to reject — \(\beta\) goes up.

The only way to reduce both \(\alpha\) and \(\beta\) at the same time is to collect more data. More data makes the test statistic more precise (smaller standard error), which makes it easier to distinguish between \(H_0\) and \(H_A\) — allowing us to keep \(\alpha\) small while also keeping \(\beta\) small. This is why sample size planning (the sample size planning chapter) is so important.

Predict → Do → Explain. Open the Decision Errors simulator — the gated activity opens with a Predict step (commit to what you expect the hit / miss / false-alarm dots to show under \(H_0\) vs. \(H_A\), at different \(n\) and \(\alpha\)), walks you through both scenarios one step at a time, and closes with an Explain step where you articulate the \(\alpha\)–\(\beta\) tradeoff in your own words. Each dot in the tool is one study: green = detected the effect (correct rejection), gray = missed the effect (Type II), red = false alarm under a true \(H_0\) (Type I). Your predictions and explanation save locally in the tool, so you can leave the activity and return to your work on the same device.

8.6 Choosing a discernibility level

The discernibility level (traditionally called the significance level) provides the cutoff for the p-value which will lead to a decision of “reject the null hypothesis.” The traditional level is 0.05. However, it is sometimes helpful to adjust the discernibility level based on the application. We may select a level that is smaller or larger than 0.05 depending on the consequences of any conclusions reached from the test.

If making a Type I error is dangerous or especially costly, we should choose a small discernibility level (e.g., 0.01 or 0.001). If we want to be very cautious about rejecting the null hypothesis, we demand very strong evidence favoring the alternative \(H_A\) before we would reject \(H_0\).

If a Type II error is relatively more dangerous or much more costly than a Type I error, then we should choose a higher discernibility level (e.g., 0.10). Here we want to be cautious about failing to reject \(H_0\) when the null is actually false.

Discernibility levels should reflect consequences of errors. The discernibility level selected for a test should reflect the real-world consequences associated with making a Type I or Type II error.

A city water department tests whether the lead content in drinking water exceeds the safe limit. They set \(H_0\): lead level \(\leq\) safe limit, and \(H_A\): lead level \(>\) safe limit. Should they use \(\alpha = 0.01\) or \(\alpha = 0.10\)?

---

A Type II error here — failing to detect dangerous lead levels — could have serious public health consequences. A Type I error — declaring the water unsafe when it is actually fine — would cause inconvenience and expense but is far less harmful than allowing people to drink contaminated water. Therefore, the water department should use a larger \(\alpha\) (like 0.10) to make it easier to detect real contamination, accepting a slightly higher chance of false alarms.

A social media company wants to test whether a new recommendation algorithm increases average time spent on the platform. They set \(H_0\): no difference, \(H_A\): the new algorithm increases engagement. Should they use \(\alpha = 0.01\) or \(\alpha = 0.10\)?

---

The stakes here are relatively low. A Type I error means rolling out an algorithm that doesn’t actually help — some wasted development effort. A Type II error means keeping the old algorithm when the new one would have been slightly better — a missed opportunity but not a disaster. A standard \(\alpha = 0.05\) or even a conservative \(\alpha = 0.01\) is reasonable, since the company can easily run another test with more data.

8.7 Looking ahead: the probability of a Type II error

We can choose \(\alpha\) directly, but \(\beta\) — the Type II error rate — depends on things we do not control: the size of the real effect, the sample size, and the variability in the data. Quantifying \(\beta\), and its complement the power of a test, is the subject of an optional later chapter on Type II error and power. There we use simulation to see how often a real effect slips past a test, and how planning the sample size brings the Type II error rate under control.

8.8 Chapter review

8.8.1 Summary

• Every hypothesis test ends in one of four outcomes, organized by the decision table: two correct decisions and two errors.
• A Type I error rejects a true \(H_0\) (a false alarm); its probability is the discernibility level \(\alpha\), which we choose.
• A Type II error fails to reject \(H_0\) when \(H_A\) is true (a missed detection); its probability is \(\beta\), which we do not directly control.
• For a fixed sample size and effect size, lowering \(\alpha\) raises \(\beta\) and vice versa — the \(\alpha\)–\(\beta\) tradeoff. Collecting more data is the way to reduce both.
• The discernibility level should be chosen to reflect the real-world consequences of each kind of error.

8.8.2 Key terms

Term	Definition
Type I error	Rejecting \(H_0\) when it is true (false positive)
Type II error	Failing to reject \(H_0\) when \(H_A\) is true (false negative)
Discernibility level (\(\alpha\))	The probability of a Type I error; chosen before the test
Type II error rate (\(\beta\))	The probability of a Type II error
Decision table	The two-by-two table of test decision versus the truth
\(\alpha\)–\(\beta\) tradeoff	At fixed \(n\), lowering one error rate raises the other

8.9 Exercises

Answers to odd-numbered exercises are provided in the Exercise Solutions appendix at the back of the book.

1. Testing for Fibromyalgia. A patient named Diana was diagnosed with Fibromyalgia, a long-term syndrome of body pain, and was prescribed anti-depressants. Being the skeptic that she is, Diana didn’t initially believe that anti-depressants would help her symptoms. However after a couple months of being on the medication she decides that the anti-depressants are working, because she feels like her symptoms are in fact getting better.

1. Write the hypotheses in words for Diana’s skeptical position when she started taking the anti-depressants.

2. What is a Type I error in this context?

3. What is a Type II error in this context?

2. Testing for food safety. A food safety inspector is called upon to investigate a restaurant with a few customer reports of poor sanitation practices. The food safety inspector uses a hypothesis testing framework to evaluate whether regulations are not being met. If he decides the restaurant is in gross violation, its license to serve food will be revoked.

1. Write the hypotheses in words.

2. What is a Type I error in this context?

3. What is a Type II error in this context?

4. Which error is more problematic for the restaurant owner? Why?

5. Which error is more problematic for the diners? Why?

6. As a diner, would you prefer that the food safety inspector requires strong evidence or very strong evidence of health concerns before revoking a restaurant’s license? Explain your reasoning.

3. True / False. Determine if the following statements are true or false, and explain your reasoning. If false, state how it could be corrected.

1. If a given value (for example, the null hypothesized value of a parameter) is within a 95% confidence interval, it will also be within a 99% confidence interval.

2. Decreasing the discernibility level (\(\alpha\)) will increase the probability of making a Type I error.

3. Suppose the null hypothesis is \(p = 0.5\) and we fail to reject \(H_0\). Under this scenario, the true population proportion is 0.5.

4. With large sample sizes, even small differences between the null value and the observed point estimate, a difference often called the effect size, will be identified as statistically discernible.

4. Online communication. A study suggests that 60% of college student spend 10 or more hours per week communicating with others online. You believe that this is incorrect and decide to collect your own sample for a hypothesis test. You randomly sample 160 students from your dorm and find that 70% spent 10 or more hours a week communicating with others online. A friend of yours, who offers to help you with the hypothesis test, comes up with the following set of hypotheses. Indicate any errors you see.

\[H_0: \hat{p} < 0.6 \quad \quad H_A: \hat{p} > 0.7\]

5. Estimating \(\pi\). In a class activity, each of 100 students experimentally estimates the value of \(\pi\), 10 separate times. Using the 10 measurements for \(\pi\) (10 values of \(\hat{\pi}\)), each student calculates a confidence interval for \(\pi\). In grading the 100 student assignments, the professor marks 7 of the assignments wrong, indicating that the 7 students must have done their experiments or analysis incorrectly because each of the 7 students reported confidence intervals that did not capture the known true value of \(\pi\), roughly 3.14159. Was the professor correct to mark the assignments wrong for having CIs that did not capture the value of 3.14159? Explain.

6. Fermenting yeast. Twenty students work individually in a biology lab to test whether using raw sucrose versus refined sugar will lead to the same yeast fermentation rate. Each student runs a full experiment independently of the other students in the lab. Of the twenty students, twelve are able to reject the null hypothesis and to claim that the fermentation rates are different.

1. Explain what type of error was likely to have occurred in this situation.
2. What change would you suggest that would lower the error rate?

7. Hypothesis statements. For each of the research claims below, fill in the value and the direction of the null and alternative hypotheses. That is, complete all aspects of the following hypothesis statements. Additionally, for each item, describe \(p\) in words.

\[H_0: p \_\_\_\_ \_\_\_\_ \quad \quad H_A: p \_\_\_\_ \_\_\_\_\]

1. On a pre-test to assess knowledge of the upcoming material, a professor wants to determine if their students know, on average, more than if they were just randomly guessing. The pre-test is 30 multiple choice questions, where each question has 5 possible responses.

2. A standard treatment is known to reduce blood pressure in 32% of patients. A clinical trial is conducted to assess whether a new medical intervention will produce results which are different than the standard treatment, in terms of the percent of patients who will have reduced blood pressure.

3. In the last presidential election 67% of registered voters turned out to vote. Will the next presidential election have a higher turn-out of voters?

StatLens Exercises

Let StatLens do the arithmetic; spend your effort on the reasoning a calculator can’t do.

1. Frame the question (and the two ways to be wrong). A pharmaceutical company tests whether a new drug raises the success rate above the existing 50% standard.

   1. Write \(H_0\) and \(H_A\) in symbols.
   2. In context, describe a Type I error and a Type II error in plain English (no Greek letters).
   3. In context, who pays the cost of each error — patients, the company, regulators? When the costs of the two errors are unequal, which one do you typically want to control more tightly, and how?

2. Pick the right \(\alpha\) for the situation. For each scenario, would you set \(\alpha\) at the conventional 0.05, lower (e.g., 0.01), or higher (e.g., 0.10)? Justify.

   1. A smoke alarm: \(H_0\) = “no fire” vs. \(H_A\) = “fire.” Type I = false alarm; Type II = silent during a real fire.
   2. Approving a new vaccine: \(H_0\) = “no effect” vs. \(H_A\) = “effective.”
   3. An A/B test of two webpage headlines: \(H_0\) = “same click rate” vs. \(H_A\) = “different.”
   4. A criminal trial: \(H_0\) = “innocent” vs. \(H_A\) = “guilty.”

3. Simulate the error rates. Open the Decision Errors simulator and step through the gated activity. Run the truth-is-\(H_0\) scenario first.

   1. Set truth = \(H_0\) (\(p = 0.50\)), \(n = 100\), \(\alpha = 0.05\). Run 500 studies. Of those, roughly what fraction reject \(H_0\)? What error does each rejection represent?
   2. Drop \(\alpha\) to 0.01 and re-run 500 studies. Did the fraction of rejections go up or down? Is this consistent with what \(\alpha\) is supposed to be?
   3. Now flip truth to \(H_A\) with \(p_{\text{true}} = 0.65\), \(n = 100\), \(\alpha = 0.05\). Run 500 studies. Roughly what fraction reject? Of the non-rejections, what kind of error are those?
   4. With truth = \(H_A\), \(p_{\text{true}} = 0.65\), \(\alpha = 0.05\), run at \(n = 50\), \(n = 100\), and \(n = 200\). How does the rejection rate (the power) change with \(n\)?

4. The conditional nature of \(\alpha\) and \(\beta\).

   1. Why is \(\alpha\) a probability conditional on \(H_0\) being true, and \(\beta\) a probability conditional on a specific \(H_A\)?
   2. A student says: “If \(\alpha = 0.05\), then 5% of rejections are wrong.” Why is this wrong, even though it sounds plausible?
   3. Suppose 1000 hypothesis tests are run, all at \(\alpha = 0.05\). In 800 of them \(H_0\) is actually true. How many false positives (Type I errors) do you expect? How does this differ from the “5% are wrong” framing?
   4. Connect to multiple comparisons: why does running 20 tests at \(\alpha = 0.05\) each give you a substantial chance of at least one false rejection, even if all 20 null hypotheses are true?

5. A glimpse of power (formal treatment in Ch 29–31). Suppose a clinical trial will test whether a drug raises the success rate from 50% to 65%. With \(n = 100\) patients and \(\alpha = 0.05\) (one-sided).

   1. Predict, without running anything: do you think this trial has enough patients to reliably detect a real 15-percentage-point effect? “Reliably” usually means at least 80% power.
   2. Open the Decision Errors simulator with truth = \(H_A\), \(p_{\text{true}} = 0.65\), \(n = 100\), \(\alpha = 0.05\) and run 1000 studies. What is the simulated power?
   3. Given the power you found, what’s the practical implication — if the drug really works, what’s the chance the trial fails?
   4. Name two things you could change before the trial begins to raise its power (say, if the effect had turned out smaller and power fell short). Which one is usually under the researcher’s control?