23 Multiple Regression

The regression models in earlier chapters used a single predictor $x$ to explain a response $y$. Most real datasets carry many potential predictors: the interest rate a lender charges depends on a borrower’s income and the loan term and their credit history; a penguin’s body mass depends on flipper length and bill length and species. Multiple linear regression extends the simple regression framework to accommodate two or more predictors at once. This chapter follows the structure of the IMS 2e chapter Inference for linear regression with multiple predictors (inf-model-mlr.qmd, CC BY-SA 3.0) — adopting IMS’s running loans case study, its coin-dish multicollinearity example, and its conditional hypothesis-testing framing — while writing the exposition anew for the simulation-first sequence and aligning the material selection to the UWL STAT 145 coursepack (Bingham, ch 18). The coursepack’s coverage of cross-validation, formal model-selection procedures, adjusted $R^2$, and the variance inflation factor is intentionally omitted or moved to Extended Content in this chapter.

See it in action. Open the Multiple Regression tool to pick a response and predictors, read the coefficient table, model $F$-test, $R^2$/adjusted $R^2$, and diagnostics (residuals, Cook’s distance). The base view shows a pairwise predictor scatterplot matrix for spotting collinearity by eye; append ?expert=true for the VIF column (Extended Content). Try possum with two predictors to reproduce this chapter’s worked example.

23.1 From one predictor to many

A single-predictor regression is a starting point, not an ending point. In the real world, a response variable is almost never driven by just one thing. When we ignore other variables that also influence the response, two things can go wrong.

Confounding can bias the slope estimate. If we regress interest rate on debt-to-income ratio alone, and borrowers with higher debt-to-income also tend to have shorter loan terms and more credit checks, the fitted slope absorbs the effects of all three. Our estimate of “how much does an extra unit of debt-to-income cost, for a comparable borrower?” is contaminated by the unmeasured differences in loan term and credit history.

Predictive accuracy suffers. Even when we don’t care about causal interpretation, ignoring available predictors leaves signal on the table. A model with three good predictors typically predicts future observations more accurately than a model with one.

Multiple regression solves the first problem by fitting all predictors together in a single model, so each coefficient represents the effect of that predictor while holding the others fixed. And it addresses the second by explaining more of the total variability with the additional predictors.

23.2 The multiple regression model

The multiple linear regression model with $k$ predictors is

\[y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k + \varepsilon,\]

where $\beta_0$ is the population intercept, $\beta_1, \dots, \beta_k$ are the population slopes for the $k$ predictors, and $\varepsilon$ is a random error term with mean 0 and constant variance $\sigma^2$.

The fitted model estimated from a sample is

\[\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \cdots + b_k x_k,\]

with $b_0, b_1, \ldots, b_k$ chosen to minimize the sum of squared residuals $\sum (y_i - \hat{y}_i)^2$ — the same least-squares criterion used in simple regression.

Geometrically, a single-predictor regression fits a line in a two-dimensional scatterplot; a two-predictor regression fits a plane in a three-dimensional cloud. With three or more predictors we lose the ability to draw the picture, but the mathematics generalizes without incident: we are always fitting the flat surface that comes closest to the observed points in the sum-of-squared-residuals sense.

23.3 Reading multiple-regression output: the loans case study

Predicting interest rates. IMS’s running case study uses loans_full_schema from the openintro R package — a curated sample of loans made through a peer-to-peer lending platform. The response is the annual interest rate charged; predictors include the borrower’s debt-to-income ratio, the loan term (36 or 60 months), and the number of credit inquiries in the past 12 months (credit_checks).

Fitting the model \[E[\texttt{interest\_rate}] = \beta_0 + \beta_1 \cdot \texttt{debt\_to\_income} + \beta_2 \cdot \texttt{term} + \beta_3 \cdot \texttt{credit\_checks}\] in software gives the following coefficient table (numbers from IMS 2e):

Predictor	Estimate	Std. Error	$t$	p-value
(Intercept)	4.31	0.20	21.9	< 0.0001
debt_to_income	0.041	0.006	6.94	< 0.0001
term	0.16	0.006	24.6	< 0.0001
credit_checks	0.25	0.020	12.7	< 0.0001

The fitted equation is

\[\widehat{\texttt{interest\_rate}} = 4.31 + 0.041 \cdot \texttt{debt\_to\_income} + 0.16 \cdot \texttt{term} + 0.25 \cdot \texttt{credit\_checks}.\]

Interpret each coefficient in context.

Intercept. A hypothetical borrower with debt-to-income 0, term 0, and 0 credit checks would receive an estimated mean interest rate of 4.31%. As is often the case, this hypothetical borrower doesn’t exist (a loan can’t have a term of 0), so the intercept mostly serves to position the fitted plane, not as a substantively meaningful prediction.
debt_to_income. Holding term and credit checks fixed, each additional unit of debt-to-income ratio is associated with a 0.041 percentage-point increase in the mean interest rate.
term. Holding debt-to-income and credit checks fixed, each additional month of loan term is associated with a 0.16 percentage-point increase in the mean interest rate. (Since term in this dataset takes only the values 36 or 60, the practical interpretation is that the 60-month loans carry a rate about $24 \times 0.16 \approx 3.8$ percentage points higher than the 36-month loans, on average.)
credit_checks. Holding debt-to-income and term fixed, each additional credit inquiry in the past 12 months is associated with a 0.25 percentage-point increase in the mean interest rate.

Each of the four $t$-statistics has a p-value below 0.0001, so each predictor contributes discernibly above and beyond the others.

23.4 Interpreting coefficients “holding others constant”

The single most important interpretive habit in multiple regression is the phrase “holding the other predictors constant.” It appears in every coefficient interpretation, and dropping it turns a correct sentence into a wrong one.

To see why, consider the loans model above. In a simple regression on debt_to_income alone, the fitted slope would absorb the effects of all three of debt-to-income, term, and credit checks — because borrowers with higher debt-to-income tend also to take out longer-term loans and to have more credit checks. The single-predictor slope estimate would be inflated by that bundle of effects.

The multiple-regression slope $b_1 = 0.041$ isolates the debt-to-income effect from the term and credit-check effects. That is why it can be interpreted as “for each additional unit of debt-to-income, holding term and credit checks fixed.” The “holding fixed” is not a philosophical qualifier; it is the only way the coefficient makes sense.

Every coefficient interpretation in multiple regression should include the phrase “holding the other predictors constant” (or an equivalent like “controlling for” or “at fixed values of”). Interpreting $b_j$ without that qualifier is one of the most common errors students make with multiple regression output.

23.5 Testing individual coefficients

Each row of the coefficient table also reports a $t$-statistic and p-value. These test the null hypothesis that a single population slope is zero — but now the hypothesis is conditioned on the other predictors being in the model.

Conditional slope hypotheses. In a multiple regression with $k$ predictors, the individual $t$-test for predictor $x_j$ tests

\[H_0: \beta_j = 0, \quad \text{given the other } k-1 \text{ predictors are included in the model.}\]

A small p-value provides evidence that $x_j$ contributes to explaining the response above and beyond what the other predictors already explain.

Rejecting $H_0: \beta_j = 0$ tells us that dropping $x_j$ from the model would meaningfully worsen the fit. Failing to reject does not tell us that $x_j$ is unrelated to $y$ — it might be strongly related, but the unique piece it contributes above what the other predictors already capture is small. This distinction matters most when predictors are correlated with each other, which is the topic of the multicollinearity section.

23.6 ANOVA for multiple regression

The ANOVA-for-regression framework introduced in the ANOVA-for-Regression chapter carries over to multiple predictors with only two changes.

For a multiple regression with $k$ predictors and $n$ observations, the ANOVA table is:

Source	df	SS	MS	$F$
Regression	$k$	$SSR$	$MSR = SSR/k$	$F = MSR/MSE$
Residual	$n - k - 1$	$SSE$	$MSE = SSE/(n-k-1)$
Total	$n - 1$	$SST$

Under $H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0$ (that no predictor contributes) and the LINE conditions, the $F$-statistic follows an $F_{k,\,n-k-1}$ distribution. A small p-value indicates that the model as a whole explains a discernible share of the variability in $y$; it does not tell us which predictors are responsible. The coefficient-level $t$-tests answer the “which one” question — with the conditional caveat above.

The interpretation of $R^2 = SSR/SST$ as “the fraction of total variability the model explains” is unchanged from simple regression. The important new fact is that adding any predictor to the model — even a completely useless one — can only increase $R^2$, never decrease it. This is what motivates the Extended Content on adjusted $R^2$ below.

Extended Content: adjusted $R^2$.

Because $R^2$ never decreases when we add predictors, comparing candidate models with different numbers of predictors purely by $R^2$ is unfair to the smaller model. Adjusted $R^2$ corrects for that unfairness:

\[R^2_{\text{adj}} = 1 - (1 - R^2) \cdot \frac{n - 1}{n - k - 1}.\]

Adding a useless predictor eats up one degree of freedom in the denominator without meaningfully reducing $SSE$, so $R^2_{\text{adj}}$ decreases rather than increases. Adjusted $R^2$ is the standard summary to compare models with different numbers of predictors; it is not part of UWL’s STAT 145 base track but is retained here for readers who encounter it in later coursework.

23.7 Multicollinearity: the coin-dish story

Two predictors in a regression exhibit multicollinearity when they are strongly linearly related to each other. This is not a problem for prediction, but it is a serious problem for interpreting individual coefficients: when two predictors carry nearly the same information, the fit cannot tell them apart, and the individual $t$-tests may fail to reject even when the joint effect of the pair is strong.

The classic teaching example (from IMS 2e, ultimately due to Jeff Witmer at Oberlin) is a coin-dish scenario. Suppose we sample 26 students and ask each to report

the total amount of money in their coin dish (dollars),
the total number of coins in the dish,
the number of low coins (coins that are not quarters).

Two simple regressions. Fitting the two simple regressions separately gives (numbers from IMS 2e):

$\widehat{\text{total\_amount}} = 0.55 + 0.13 \cdot \text{number\_of\_coins}$ — slope $b_1 = 0.13$, p-value very small.
$\widehat{\text{total\_amount}} = 2.28 + 0.02 \cdot \text{number\_of\_low\_coins}$ — slope $b_1 = 0.02$, p-value large (no discernible relationship).

Interpret the two slopes and, briefly, why they take the values they do.

For every additional coin in the dish, the total amount goes up by about $0.13 on average — roughly the average value of a US coin, as we’d hope. For every additional low coin (a penny, nickel, or dime), the total goes up by only about $0.02 on average — which sounds reasonable but is barely discernible from zero, because adding a low coin only adds pennies, nickels, or dimes.

Now fit the model with both predictors together. IMS reports

\[\widehat{\text{total\_amount}} = 0.02 + 0.21 \cdot \text{number\_of\_coins} - 0.16 \cdot \text{number\_of\_low\_coins},\]

both coefficients discernible.

Explain what happened. Why is the coefficient on number_of_low_coins negative in the multi-predictor model when it was slightly positive on its own?

The reason is the “holding others constant” language. In the multi-predictor model,

$b_1 = 0.21$ says: for each additional coin while the number of low coins stays constant, the total goes up by 21 cents. Increasing “total coins” while keeping “low coins” fixed means adding a quarter — which adds 25 cents on average. So the coefficient makes sense.
$b_2 = -0.16$ says: for each additional low coin while the total number of coins stays constant, the total goes down by 16 cents. Swapping a quarter (25 cents) for a low coin (a few cents) drops the total by roughly that amount. So this coefficient also makes sense.

The two individual slopes are answering different questions because they condition on the other variable in different ways. When predictors are strongly related to each other, the multi-predictor coefficients can look very different from the simple-regression slopes — and both interpretations are correct, as long as we honor the “holding others constant” qualifier.

A telltale sign of multicollinearity is a predictor whose relationship to the response looks strong on a scatterplot yet has an individual $t$-test that fails to reject. The predictor may be redundant with another predictor already in the model, contributing no additional information once the correlated predictor is accounted for.

The most direct diagnostic is to plot the predictors against each other. A strong linear pattern between two predictors is evidence that they carry overlapping information. Common responses to strong pairwise associations: (i) drop one of the collinear predictors, (ii) combine them into a single summary variable (an average or sum, when units allow), or (iii) accept that individual coefficients cannot be separately interpreted but that the model still predicts well as a whole.

Extended Content: the variance inflation factor.

Beyond scatterplots, statisticians quantify the severity of multicollinearity with a single number per predictor: the variance inflation factor (VIF). For predictor $x_j$,

\[VIF_j = \frac{1}{1 - R^2_j},\]

where $R^2_j$ is the $R^2$ of a regression of $x_j$ on all the other predictors. A predictor uncorrelated with the others has $R^2_j = 0$ and $VIF_j = 1$; a predictor perfectly explained by the others has $R^2_j = 1$ and infinite VIF. Rules of thumb (not sharp thresholds): $VIF < 5$ is fine; $5 \le VIF < 10$ warrants caution; $VIF \ge 10$ means individual coefficients are unreliable.

VIF is retained here for readers, instructors, and later coursework that want a formal metric; UWL’s STAT 145 sequence detects multicollinearity through the scatterplot approach above and does not require the VIF machinery.

23.8 Categorical predictors: a brief note

The models above use numeric predictors only. Multiple regression also accommodates categorical predictors by encoding them as indicator variables (also called dummy variables). For a two-level categorical variable (say, verified_income = yes or no), we create a numeric variable that takes the value 1 when the category is “yes” and 0 otherwise; then this indicator enters the regression like any other numeric predictor. The coefficient on the indicator represents the difference in mean response between the two categories, holding the other predictors fixed. For categorical variables with three or more levels, the same idea applies but requires one indicator variable per non-baseline category.

A full treatment of indicator variables — including baseline choice, interpretation, and interaction terms — is beyond the scope of this chapter and belongs in a follow-on course. IMS 2e’s inf-model-mlr.qmd chapter covers the extension in detail if you want the deeper version.

23.9 Diagnostics for multiple regression

The LINE conditions from the linear regression chapter carry over unchanged to multiple regression, but the checks look slightly different because we no longer have a single predictor’s scatterplot to eyeball.

Linearity. Plot each predictor against the residuals. A curved pattern in one plot suggests that predictor’s relationship with the response is not linear at fixed values of the others.

Independence. Same as in simple regression: the design of the study earns this assumption. Random sampling or random assignment protects independence.

Normality of residuals. A residual histogram or Q-Q plot works the same as in simple regression.

Equal variance. The residual-versus-fitted plot is still the go-to diagnostic. A fan shape suggests the residual spread grows with $\hat{y}$; if so, coefficient standard errors are unreliable and the reported CIs are too narrow.

Add one MLR-specific check:

Multicollinearity. Inspect scatterplots of each pair of predictors. Strong pairwise associations flag pairs that carry overlapping information about the response. (Extended-content readers can also inspect the variance inflation factor for a formal quantitative check.)

23.10 Chapter review

23.10.1 Summary

Multiple regression extends the simple regression model to two or more predictors, fit by least squares. The interpretation of each coefficient becomes “the mean change in $y$ per unit change in $x_j$ holding the other predictors fixed” — a qualifier that separates multiple regression’s slope from the simple-regression slope that would absorb the other predictors’ effects. Software reports coefficient estimates alongside a coefficient table, an ANOVA table with a model-level $F$-test, and the goodness-of-fit statistic $R^2$. Individual $t$-tests for each coefficient are conditional on the other predictors being in the model. Multicollinearity — overlapping information between predictors — is a new concern that only arises with multiple predictors; the standard check is a scatterplot of each pair of predictors, and the coin-dish example illustrates why interpretations of “simple” and “multiple” slopes can look wildly different when predictors are correlated. Diagnostics for the LINE conditions carry over from simple regression. Categorical predictors extend the framework through indicator variables and are covered in follow-on courses.

23.10.2 Terms

Multiple linear regression, predictor, response, coefficient table, conditional hypothesis test, “holding others constant,” model-level $F$-test, multicollinearity, indicator variable, residual-versus-fitted plot, extrapolation in multi-dimensional predictor space. (Extended-content only: adjusted $R^2$, variance inflation factor / VIF.)

23.10.3 Attribution

This chapter is a remix of the IMS 2e chapter Inference for linear regression with multiple predictors (inf-model-mlr.qmd, Cetinkaya-Rundel and Hardin, CC BY-SA 3.0). The loans case study, the coin-dish multicollinearity example, and the conditional hypothesis-testing framing are IMS’s; the exposition is rewritten. IMS’s sections on cross-validation, formal model-selection procedures, adjusted $R^2$, and VIF are omitted or moved to Extended Content in alignment with UWL’s STAT 145 coursepack (Bingham, coursepack/sections/Unit4/ch18-additional-regression.qmd). Readers who want IMS’s cross-validation treatment or the full indicator-variable extension should consult the IMS source directly.

23.11 Exercises

Answers to odd-numbered exercises are provided in the Exercise Solutions appendix at the back of the book.

Note. Exercises 3 (VIF interpretation) and 6 (adjusted $R^2$) draw on the Extended Content callouts in the chapter. Skip them if you are following the base track.

Reading the loans coefficient table. Recall the loans model from the chapter, predicting interest rate from debt-to-income, loan term (in months), and credit checks in the past 12 months:

\[\widehat{\texttt{interest\_rate}} = 4.31 + 0.041 \cdot \texttt{debt\_to\_income} + 0.16 \cdot \texttt{term} + 0.25 \cdot \texttt{credit\_checks}.\]
1. Predict the interest rate for a borrower with debt-to-income 15, a 36-month loan, and 2 credit checks in the last year.
2. Interpret the coefficient on credit_checks in context. Include the phrase “holding the other predictors constant.”
3. The individual $t$-test on credit_checks has a very small p-value. State the null hypothesis this test evaluates, in symbols and in words, including the “given other predictors are in the model” qualifier.
Simple vs. multiple regression slope. Suppose the simple regression of interest rate on credit_checks alone gives a slope of 0.60 — larger than the multi-predictor slope of 0.25.
1. Why is the simple-regression slope larger?
2. Which slope answers the question “for two borrowers with the same debt-to-income and same loan term, how much does one extra credit check cost the borrower with more inquiries?”
3. Which slope answers the question “for two random borrowers differing only in credit-check count (with all their other characteristics also differing on average), how different are their expected interest rates?”
VIF interpretation (Extended Content). A multiple regression on employee salary reports the following VIFs:

Predictor VIF

Age 1.4

Years of experience 8.7

Years at company 9.1

Job level 1.2
1. Which two predictors are causing concern about multicollinearity, and why?
2. Explain in one sentence why high VIF makes it hard to interpret those two coefficients individually.
3. Name two possible responses to the high VIFs.
The coin-dish story. Recall the coin-dish example from the chapter. The two simple regressions give slopes of 0.13 (on total number of coins) and 0.02 (on number of low coins). The multiple regression with both predictors gives slopes of 0.21 (coins) and $-0.16$ (low coins).
1. In the multi-predictor model, the slope on number_of_low_coins is negative. Explain in one sentence why this makes sense in context, using the “holding the other predictor fixed” framing.
2. A student says: “The simple regression shows low coins have a positive effect on total amount, but the multiple regression flips it to negative. One of them must be wrong.” Push back on this.
3. When two predictors are strongly related to each other, the “same predictor” can appear to have a positive marginal effect and a negative conditional effect. What does this tell you about the danger of interpreting single-predictor slopes as if they were “the effect” of the predictor?
When to add a predictor. A researcher must choose whether to add a fifth predictor to a four-predictor model of house prices. The new predictor’s individual $t$-test has p-value 0.28, and a scatterplot shows a moderately strong linear relationship between the new predictor and one of the four already in the model.
1. Should the researcher add the predictor purely based on the statistical evidence?
2. Suppose the researcher has a strong subject-matter reason to include the predictor even though its p-value is large. Is that a valid reason to include it?
3. The residual standard error (root MSE) after adding the predictor is essentially unchanged. What does this tell you about the predictor’s contribution?
Adjusted $R^2$ vs. $R^2$ (Extended Content). Two candidate models are fit to the same data ($n = 60$):
- Model A (3 predictors): $R^2 = 0.62$, adjusted $R^2 = 0.60$.
- Model B (5 predictors): $R^2 = 0.65$, adjusted $R^2 = 0.59$.
1. Which model has more explained variability by raw $R^2$?
2. Which model is preferred by adjusted $R^2$? Why do they disagree?
3. Argue in one sentence why adjusted $R^2$ is the more honest comparison when the models have different numbers of predictors.

Predictor	VIF
Age	1.4
Years of experience	8.7
Years at company	9.1
Job level	1.2

StatLens Exercises

Let StatLens do the arithmetic; spend your effort on the reasoning a calculator can’t do.

Note. Exercises 4 and 5 optionally use the Extended Content variance inflation factor material, available via expert mode on the Multiple Regression tool. The reasoning still works if you substitute “scatterplot showing a strong relationship between predictors” for high VIF and “no strong relationship on the predictor scatterplot” for low VIF.

Frame the loans question. A lender wants to build a model to predict a borrower’s interest rate from debt_to_income, term, and credit_checks — the same three predictors the chapter uses.
1. Write the multiple regression model in the standard form $y = \beta_0 + \beta_1 x_1 + \cdots$ with all variables clearly labeled.
2. State the null and alternative hypotheses for the model-level $F$-test in words.
3. State the null hypothesis for the individual $t$-test on debt_to_income — including the “given other predictors are in the model” qualifier. Why is this a different question from the one the $F$-test asks?
Choose predictors thoughtfully. For each scenario, decide whether to include the new predictor in the model. Options: include, exclude, or it depends (and specify the deciding factor).
1. The new predictor’s individual $t$-test has p = 0.03; its scatterplot vs. existing predictors shows no strong relationships; residual standard error drops noticeably after adding it.
2. The new predictor’s $t$-test has p = 0.42; residual standard error is essentially unchanged.
3. The new predictor’s $t$-test has p < 0.001; its scatterplot vs. an existing predictor shows a very strong linear relationship (or, equivalently, its VIF is 14.2).
4. The new predictor has a strong scientific reason to be in the model (a well-established causal driver), $t$-test p = 0.20, no strong scatterplot associations with the other predictors.
Read the coefficient table (possum data). Open the Multiple Regression tool loaded with the possum data. Select head length as the response and both total length and skull width as predictors.
1. Write the fitted equation with numeric coefficients.
2. Interpret each slope in context, using the phrase “holding the other constant.”
3. Compare the total-length slope in the two-predictor model to the total-length slope from the simple regression in the linear-regression chapter. Are they the same? If they differ, what does that tell you about what the simple slope was actually measuring?
4. Report the model-level $F$-statistic and its p-value. Does the model contribute discernibly?
Multicollinearity via scatterplot (Extended: VIF). Add a third predictor (say, tail length) to the model in Exercise 3.
1. Open the pairwise scatterplot matrix of the three predictors. Are any pair of predictors visibly related? Which pair?
2. Report the coefficient for total length and its standard error. How does the standard error compare to the two-predictor version? What does the change reflect?
3. Optional (Extended Content, requires expert mode): Report the VIFs for all three predictors. Do the VIF values agree with what the pairwise scatterplot showed?
4. A student says: “adjusted $R^2$ went up, so the third predictor is clearly worth including.” Is that always sufficient justification? What else should you check before finalizing the model?
When $F$ and $t$ disagree. Use the Multiple Regression tool to construct or find a dataset where the model-level $F$-test rejects at $\alpha = 0.05$ but none of the individual $t$-tests do.
1. Report the $F$-statistic, its p-value, and the p-values of the individual $t$-tests. What does this pattern suggest?
2. Inspect the pairwise scatterplot matrix of the predictors. Are any pairs strongly related? (Optional Extended Content: report the VIFs.)
3. What does “the model as a whole works, but no individual predictor is discernible” mean for the researcher trying to use this model?
4. A student says: “If no individual $t$-test rejects, we should throw out the model.” Push back on that argument in one sentence.

Source	df	SS	MS	\(F\)
Regression	\(k\)	\(SSR\)	\(MSR = SSR/k\)	\(F = MSR/MSE\)
Residual	\(n - k - 1\)	\(SSE\)	\(MSE = SSE/(n-k-1)\)
Total	\(n - 1\)	\(SST\)