Skip to main content

Correlation and Linear Regression

From covariance to r to a fitted line, then check the residuals and find the point that moves the fit

Every test so far compared means across groups. Today both variables are continuous. Correlation asks whether two variables are linearly associated. Regression asks how much one changes per unit of the other, and predicts one from the other. Same recipe: H₀ as math, α first, look, check, test, conclude.

Lecture A012 used Iris versicolor. Here you use Adelie penguins: does flipper length go with body mass? Same workflow, so the logic transfers even though the numbers differ.

Correlation Regression
Question Are X and Y linearly associated? How much does Y change per unit X?
Symmetry Swap X and Y, same r Y is predicted from X; swapping changes the line
Output One number, r, with a p-value and CI A line: intercept and slope, each with a p-value
Prediction None Y for a new X
H₀ true correlation = 0 true slope = 0
R cor(), cor.test() lm(), summary()

Correlation is the right tool when neither variable is the cause, as with two exam scores. Regression is the right tool when one variable is the explanatory one, as with a dose you set and a response you measure.

Function What it does Example
cov(x, y) Covariance: do X and Y move together? Sign is meaningful, size depends on units cov(penguin_data$flipper_length_mm, penguin_data$body_mass_g)
cor(x, y) Pearson’s r: covariance divided by both standard deviations, so it runs from −1 to 1 cor(x, y)
cor.test(x, y) r with a p-value and 95% CI; H₀ is true correlation = 0 cor.test(x, y)
lm(y ~ x, data = ) Fits the least-squares line lm(body_mass_g ~ flipper_length_mm, data = penguin_data)
summary(model) Intercept, slope, their p-values, and R² summary(lm_model)
coef(model) Just the intercept and slope coef(lm_model)
residuals(model) Observed minus predicted, one per row shapiro.test(residuals(lm_model))
plot(model, which = 1) Residuals vs fitted: linearity and equal spread
plot(model, which = 4) Cook’s distance per observation: which points move the fit
cooks.distance(model) The numbers behind that plot max(cooks.distance(lm_model))
geom_smooth(method = "lm") Draws the fitted line and its 95% band on a scatterplot

cor.test() returns the same kind of object as t.test(), so $estimate, $p.value, and $conf.int work. summary(lm_model)$r.squared pulls R² out of a model summary.

Part A: Look first

1. Load the packages

NoteHint

Load tidyverse (dplyr and ggplot2) and palmerpenguins (the penguins table).

TipSolution
library(tidyverse)
library(palmerpenguins)

2. All 344 penguins, three species

Plot first, and look for groups. Make a scatterplot of body_mass_g (y) against flipper_length_mm (x) for every penguin, with color = species inside aes(). Save it as scatter_all and display it.

NoteHint

Add geom_point() with +. The two penguins with missing measurements are dropped with a warning, which is fine.

TipSolution
scatter_all <- ggplot(penguins, aes(x = flipper_length_mm, y = body_mass_g, color = species)) +
  geom_point()
scatter_all

Three clusters. Gentoo sit high and right, Adelie low and left, Chinstrap in between. A line through all 344 would mostly be measuring the gap between species, not flipper length. So, as in lecture, pick one species.

3. One species

Keep Adelie penguins with non-missing body_mass_g and flipper_length_mm. Save the result as penguin_data.

NoteHint

Three conditions inside one filter(), separated by commas: the species, and !is.na() on each of the two columns.

TipSolution
penguin_data <- penguins |>
  filter(species == "Adelie", !is.na(body_mass_g), !is.na(flipper_length_mm))
penguin_data

151 Adelie penguins with both measurements.

  1. Which variable is X (explanatory) and which is Y (response) in “does flipper length predict body mass”?
  2. Would swapping them change r? Would it change the regression line?
  3. Why fit within one species instead of all 344 penguins?

Answers: flipper length is X, body mass is Y. Swapping leaves r unchanged but gives a different line, because regression minimizes vertical distances to Y. Pooling species would make the line measure the species gap rather than the within-species relationship, like the pooled sepal plot in lecture where r flipped from +0.53 to −0.12.

4. Scatterplot with the fitted line

Plot body_mass_g against flipper_length_mm for penguin_data, add geom_point(), then add the fitted line with geom_smooth(method = "lm"). Save it as scatter_lm and display it.

NoteHint

Two layers. geom_smooth(method = "lm") fits a straight line; without method = "lm" it draws a wiggly curve instead.

ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point() +
  geom_smooth(method = "lm")
TipSolution
scatter_lm <- ggplot(penguin_data, aes(x = flipper_length_mm, y = body_mass_g)) +
  geom_point() +
  geom_smooth(method = "lm")
scatter_lm

A rising cloud with plenty of scatter. The grey band is the 95% confidence band for the line. This is the figure Coding Assignment 4 asks for.

Part B: Correlation

Research question: Are flipper length and body mass linearly associated in Adelie penguins?

H₀: the true correlation is 0. Hₐ: it is not 0. Pearson correlation, α = 0.05.

5. Covariance depends on the ruler

Covariance measures whether X and Y move together: each penguin contributes (x − x̄)(y − ȳ), and the average of those products is the covariance. Compute it for flipper length and body mass and save it as cov_raw. Then compute it again with flipper length in centimeters (divide by 10) and body mass in kilograms (divide by 1000), and save that as cov_rescaled. Print both.

NoteHint

cov() takes two vectors. You can divide a column right inside the call.

cov(mtcars$wt, mtcars$mpg)
cov(mtcars$wt * 1000, mtcars$mpg)
TipSolution
cov_raw <- cov(penguin_data$flipper_length_mm, penguin_data$body_mass_g)
cov_rescaled <- cov(penguin_data$flipper_length_mm / 10, penguin_data$body_mass_g / 1000)
cov_raw
cov_rescaled

1404 mm·g versus 0.14 cm·kg. Same penguins, same picture, a number 10,000 times smaller. The sign is useful; the size depends on the units. That is why we need r.

6. From covariance to r

Dividing the covariance by both standard deviations removes the units: r = cov(x, y) / (sₓ · sᵧ). Compute it that way and save it as r_by_hand. Then compute it with cor() and save it as r_cor. Print both.

NoteHint

You already have cov_raw. Divide it by sd() of flipper length times sd() of body mass. cor() takes the same two vectors cov() did.

TipSolution
r_by_hand <- cov_raw / (sd(penguin_data$flipper_length_mm) * sd(penguin_data$body_mass_g))
r_cor <- cor(penguin_data$flipper_length_mm, penguin_data$body_mass_g)
r_by_hand
r_cor

Both are 0.468. Positive, moderate. Try the same division with cov_rescaled and the rescaled standard deviations: still 0.468. r does not care about the ruler.

7. r needs a p-value

r = 0.47 from 151 penguins is not the same evidence as r = 0.47 from 8. Run cor.test() on flipper length and body mass. Save it as cor_result and print it.

NoteHint

Same two vectors you gave cor(). The output has the shape of a t.test() result: a statistic, df, a p-value, a confidence interval, and the estimate.

TipSolution
cor_result <- cor.test(penguin_data$flipper_length_mm, penguin_data$body_mass_g)
cor_result

r = 0.47, t = 6.47, df = 149, p = 1.3 × 10⁻⁹, 95% CI 0.33 to 0.58. The interval does not include 0.

8. Make the call

Are flipper length and body mass associated in Adelie penguins at α = 0.05? Assign flipper_mass_associated to TRUE or FALSE.

NoteHint

Compare cor_result$p.value with 0.05, and check whether cor_result$conf.int contains 0.

TipSolution
flipper_mass_associated <- TRUE

p is far below 0.05 and the CI for r excludes 0. Reject H₀: a moderate positive association. Not “flipper length causes body mass”; nothing was manipulated.

Three studies all report r = 0.30. One has n = 8, one n = 30, one n = 300. Which is the strongest evidence that the true correlation is not zero?

Answer: n = 300 (p ≈ 10⁻⁷). With n = 8 the same r gives p = 0.47; with n = 30, p = 0.11. r measures strength in the sample; p measures how surprising that r would be if the true value were 0, and that depends on n.

Part C: Linear regression

Research question: How much heavier is an Adelie penguin per extra millimeter of flipper?

H₀: the true slope is 0. Hₐ: it is not 0. Least-squares regression of body mass on flipper length, α = 0.05.

9. Fit the line

Fit lm() with body_mass_g as the response and flipper_length_mm as the explanatory variable. Save the model as lm_model and print summary(lm_model).

NoteHint

Same formula shape as aov(): response, tilde, explanatory variable, data =. Response goes on the left.

fit <- lm(mpg ~ wt, data = mtcars)
summary(fit)
TipSolution
lm_model <- lm(body_mass_g ~ flipper_length_mm, data = penguin_data)
summary(lm_model)

Slope 32.8 g per mm, p = 1.3 × 10⁻⁹. Intercept −2536 g. Multiple R-squared 0.219. The slope’s p-value is the same as cor.test() gave: with one predictor they are the same test.

  1. The slope is 32.8. What does it mean?
  2. The intercept is −2536 g. What should you make of it?

Answers: (1) Each extra millimeter of flipper goes with about 33 g more body mass, on average. Not “mass equals 33 times flipper length”, and not “flipper length explains 33% of mass”. (2) It is the predicted mass of a penguin with a 0 mm flipper. Flippers in this sample run from 172 to 210 mm, so X = 0 is far outside the data and the intercept has no biological meaning. It is just where the line crosses the axis. The model is still fine.

10. R² by hand

R² is the fraction of the scatter in Y that the line accounts for. Compute the sum of squared distances from each body mass to the mean body mass and save it as ss_mean. Compute the sum of squared residuals and save it as ss_line. Then compute r_squared_by_hand as 1 − ss_line / ss_mean. Print all three.

NoteHint

A sum of squares is sum((values)^2). For ss_mean the values are body_mass_g - mean(body_mass_g). For ss_line they are residuals(lm_model), which are already observed minus predicted.

TipSolution
ss_mean <- sum((penguin_data$body_mass_g - mean(penguin_data$body_mass_g))^2)
ss_line <- sum(residuals(lm_model)^2)
r_squared_by_hand <- 1 - ss_line / ss_mean
ss_mean
ss_line
r_squared_by_hand

Scatter around the mean is 31.5 million g²; around the line it is 24.6 million. The line removed 22% of the scatter, so R² = 0.219. And 0.468² = 0.219: with one predictor, R² is r squared.

11. R² from the summary

Pull R² out of summary(lm_model) and save it as lm_r_squared. Print it, then confirm it equals r_cor^2.

NoteHint

summary(lm_model) is a list; $r.squared is one of its pieces.

TipSolution
lm_r_squared <- summary(lm_model)$r.squared
lm_r_squared
r_cor^2

0.219 both ways. Flipper length accounts for about a fifth of the variation in Adelie body mass. The other four fifths is everything else: sex, year, condition, measurement error.

12. Residuals vs fitted

Regression makes four assumptions: a linear relationship, independent observations, normal residuals, and equal spread of residuals along the line. Independence comes from the design (each penguin measured once). The other three are checked on the residuals, after the fit. Run the block: plot(lm_model, which = 1) draws each residual against its fitted value.

NoteHint

You are looking for two things: no curve (linearity) and no fan (equal spread). A shapeless cloud around the dashed zero line is what you want.

TipSolution

A shapeless cloud. The red trend line wobbles but does not curve, and the spread is about the same at small and large fitted values. Linearity and equal spread look fine. R labels the three largest residuals by row number; that is information, not an instruction to delete them.

13. Shapiro-Wilk on the residuals

Run shapiro.test() on residuals(lm_model). Save it as shapiro_resid and print it.

NoteHint

Same call as Coding 9, step 8, with the regression model in place of the ANOVA model. Not on body_mass_g, not on flipper_length_mm.

TipSolution
shapiro_resid <- shapiro.test(residuals(lm_model))
shapiro_resid

W = 0.988, p = 0.23. No evidence against normal residuals.

14. Make the call

Two logicals. assumptions_ok: do the residual plot and Shapiro test support the model? slope_significant: is the slope different from 0 at α = 0.05?

NoteHint

The slope’s p-value is in the flipper_length_mm row of summary(lm_model), under Pr(>|t|).

TipSolution
assumptions_ok <- TRUE
slope_significant <- TRUE

Shapeless residual plot, Shapiro p = 0.23, slope p = 1.3 × 10⁻⁹. Each extra millimeter of flipper goes with 33 g more body mass, and flipper length accounts for 22% of the variation in mass. Significant and modest at the same time.

Study 1: R² = 0.02, p < 0.001, n = 3000. Study 2: R² = 0.85, p = 0.08, n = 5. Which statement is best?

  1. Study 1 found the stronger relationship
  2. Study 2 found the stronger relationship
  3. Study 2 found a significant relationship
  4. Neither study found a relationship

Answer: B. R² is strength; p is evidence. Study 2’s relationship is far stronger, but five points are not enough to rule out chance. Study 1’s is certain to exist and almost too small to matter. Report both numbers, always.

Part D: One point can rotate the line

A regression line is pulled toward every point, but not equally. A point far out along X has extra leverage: it can rotate the whole line. The Adelie data has no such point, so this part uses a small made-up dataset of 31 points, the same demonstration as in lecture A012. You will see the point, measure how much it moves the slope, and decide what to do about it.

15. Look for a far-out point

The block below builds a small dataset, lev_plus, and draws it with the fitted line. Nothing to write; run it and look. Is there a point that sits apart from the rest?

NoteHint

Thirty points run from X = 0 to 10. Where is the thirty-first?

TipSolution

One point sits at X = 22, Y = 4, far to the right of the others, with nothing between X = 10 and X = 20. The fitted line tilts down toward it and no longer runs through the cloud of 30. Flagging it on the plot is the first step. Next, measure what it does.

16. Fit the line twice

How much does that one point change the slope? Fit the line with all 31 points and save it as lm_lev. Then keep only the rows with x < 20, save them as lev_without, and fit the same model on them as lm_without. Pull the slope out of each with coef() and save the two slopes as slope_with and slope_without. Print both slopes.

NoteHint

coef() returns two numbers, the intercept and the slope. The slope is the second one.

fit <- lm(mpg ~ wt, data = mtcars)
coef(fit)
coef(fit)[2]

filter(x < 20) keeps the 30 ordinary points.

TipSolution
lm_lev <- lm(y ~ x, data = lev_plus)
lev_without <- lev_plus |>
  filter(x < 20)
lm_without <- lm(y ~ x, data = lev_without)
slope_with <- coef(lm_lev)[2]
slope_without <- coef(lm_without)[2]
slope_with
slope_without

Slope 0.40 with the point, 0.92 without it. One observation out of 31 cut the slope by more than half. To see both lines on the same plot, add a second geom_smooth() that uses only the 30 ordinary points:

ggplot(lev_plus, aes(x = x, y = y)) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE, linetype = "dashed") +
  geom_smooth(data = lev_without, method = "lm", se = FALSE)

17. Cook’s distance

Cook’s distance puts a number on what you just saw: for each observation, how much the fitted line would move if that observation were removed. Run plot(lm_lev, which = 4). Then compute the rule-of-thumb cutoff cook_cut as 4 divided by the number of rows, and max_cook as the largest value of cooks.distance(lm_lev). Print both.

NoteHint

nrow(lev_plus) is the number of observations. cooks.distance() returns one number per row; max() picks the largest.

TipSolution
plot(lm_lev, which = 4)
cook_cut <- 4 / nrow(lev_plus)
max_cook <- max(cooks.distance(lm_lev))
cook_cut
max_cook

One bar per observation; R labels the three largest. Observation 31 has Cook’s distance 11, about 85 times the 4/n cutoff of 0.13. Every other bar is below the cutoff. One point is doing all the work, and this plot finds it even when the scatterplot is too crowded to see it.

18. Make the call

The far-out point is not impossible and there is no reason to think it is a data-entry error. Should you delete it? Assign delete_the_point to TRUE or FALSE.

NoteHint

Same questions as patient 9 in Coding 8. Impossible? Entry error? If neither, what do you do instead of deleting?

TipSolution
delete_the_point <- FALSE

Cook’s distance flags influence, not error. Keep the point, report both slopes (0.40 with it, 0.92 without), and say that the answer depends on one observation with no data between X = 10 and 20. Dropping it because it moves the line is choosing the answer you wanted.

19. Recap: functions you practiced

Function Role
ggplot() + geom_point() with color = species Plot first; look for groups
filter() One species, no missing values
geom_smooth(method = "lm") Fitted line and its confidence band
cov(x, y) Direction of association; size depends on units
cor(x, y) r: covariance over both standard deviations, −1 to 1
cor.test(x, y) r with p-value and CI; H₀: true correlation = 0
lm(y ~ x, data = ) Least-squares line
summary(lm_model) Intercept, slope, p-values, R²
sum((y - mean(y))^2) and sum(residuals(model)^2) SS around the mean and around the line
summary(lm_model)$r.squared R² = 1 − SS_line / SS_mean = r²
plot(model, which = 1) Residuals vs fitted: linearity and equal spread
shapiro.test(residuals(model)) Normality of residuals
coef(model)[2] The slope, to compare fits with and without a point
plot(model, which = 4) and cooks.distance(model) Which observation moves the fit
4 / nrow(data) Rule-of-thumb Cook’s distance cutoff

If you can explain each row in your own words, you have covered Coding Assignment 4, Part D and lecture A012.

ImportantThree things to say in your write-up
  1. Slope, p, and R² are three different facts. “33 g per mm (p = 1.3 × 10⁻⁹, R² = 0.22)” says how much, how sure, and how much is explained. A tiny p with a tiny R² is a real but weak relationship. A big R² with p = 0.08 is a strong pattern you cannot yet trust.
  2. Association is not cause. Nothing was manipulated. Longer flippers go with heavier bodies; neither causes the other.
  3. Report influence honestly. If one point moves the slope, give the fit with it and without it, say where the gap in X is, and let the reader see that the conclusion rests on one observation. Do not delete it to make the line behave.

Keep playing

What pooling would have done. Fit the penguin regression on all three species together.

R² = 0.76 pooled, 0.22 within Adelie. Most of the pooled “prediction” is the species gap you saw in step 2. The lecture’s iris sepal slide showed the same thing: pool the species and the relationship changes.

Pooling can flip the sign. Within Adelie, bill length and bill depth are positively correlated. Across all three species, the correlation is negative.

r = +0.39 within Adelie, −0.24 pooled. Gentoo have long, shallow bills, which drags the pooled line the other way. Plot first, look for groups.

Which analysis? Match each design to a test.

  1. Exam 1 and exam 2 scores for every student: are the two related?
  2. 1000 adults: does blood pressure predict blood glucose?
  3. 30 mice assigned to five caffeine doses; heart rate measured.
  4. 60 bacterial colonies: diameter measured, each scored resistant or not. Does resistance relate to size?

Answers: (1) correlation; neither exam explains the other. (2) regression; one variable is named as the predictor. (3) regression; dose was set by the experimenter, so it is the explanatory variable, and the five doses are numbers on a scale, not just labels. (4) Welch t-test; one continuous variable (diameter) compared between two groups (resistant or not).