Check Assumptions Before a Parametric Test

Clean data, flag outliers, assess normality, and choose the right transform — all in your browser

Welcome back. Everything on this page runs real R in your browser — same setup as the other coding activities. Before you run a parametric test like a t-test, you need to ask whether your data meet the test’s assumptions — especially normality. Today you will walk through the assumption-checking pipeline from lecture: handle missing values, remove impossible entries, flag distributional outliers, assess normality with a histogram and Shapiro-Wilk test, and try transformations when the raw scale fails. You will not run the t-test itself.

The first time you click Run Code, your browser downloads R once (a few seconds). After that it is quick.

1. Load the tidyverse

Coding Assignment 3 starts the same way. Run library(tidyverse) to load dplyr, ggplot2, and friends. A conflicts message afterward is expected, not an error.

NoteHint

From ?library: library(package, ...) attaches a package so its functions become available. Replace package with the name of the collection you need.

Try ?library in R for the full help page.

TipSolution 
library(tidyverse)

On DataHub and in your graded assignment you will type exactly this.

2. Build the study group

A one-sample t-test compares one group to a reference value — so you need a single species, not the whole dataset mixed together. Filter to Adelie penguins measured in 2007, then create a new column bill_area as the product of bill length and bill depth (mm²). Save the result as bill_data.

NoteHint

From ?filter: keep rows where conditions are TRUE. From ?mutate: add or change columns.

penguins |>
  filter(species == "Gentoo") |>
  mutate(ratio = bill_length_mm / bill_depth_mm)

Filter to Adelie and year 2007, then multiply the two bill columns.

TipSolution 
bill_data <- penguins |>
  filter(species == "Adelie", year == 2007) |>
  mutate(bill_area = bill_length_mm * bill_depth_mm)

You should have 50 rows — one Adelie cohort from a single field season.

3. Glimpse the data

Run glimpse() on bill_data to see row count, column names, and types.

NoteHint

From ?glimpse: glimpse(x) prints rows, columns, and types.

glimpse(mtcars)

Apply it to the dataset you just built.

TipSolution 
glimpse(bill_data)

You should see 50 rows. bill_area is <dbl> — and one penguin has missing bill measurements, so some rows will show NA for area.

4. Warm-up — what does “normal enough” look like?

Before working with bill_area, practice on bill length in the same birds — a single measurement that already looks fairly symmetric. Make a histogram of bill_length_mm with ggplot() and geom_histogram().

NoteHint

From ?ggplot: add a layer with +.

ggplot(mtcars, aes(x = mpg)) +
  geom_histogram()

Map bill length to the x-axis and add geom_histogram().

TipSolution 
ggplot(bill_data, aes(x = bill_length_mm)) +
  geom_histogram()

The distribution looks roughly bell-shaped — a good starting point for a normality check.

5. Run Shapiro-Wilk on bill length

Run shapiro.test() on the non-missing bill-length values in bill_data. Save the result as shapiro_length and print it.

NoteReading Shapiro-Wilk output

The null hypothesis is normality. A small p-value (conventionally below 0.05) means the data look inconsistent with a normal distribution. A large p-value means you do not have evidence against normality — not proof that the data are perfectly normal.

NoteHint

From ?shapiro.test: pass a numeric vector directly.

shapiro.test(mtcars$mpg)

Use the bill-length column from bill_data.

TipSolution 
shapiro_length <- shapiro.test(bill_data$bill_length_mm)
shapiro_length

You should see p ≈ 0.69 — no evidence against normality for bill length.

6. Mean bill area with missing values

A published survey of a neighboring Adelie population reported a mean bill area of 700 mm². Calculate the mean bill_area across all penguins in your sample and save it as mean_raw. Use na.rm = TRUE. Print the result and compare it to the reference value.

NoteHint

From ?mean: mean(x, na.rm = FALSE) — set na.rm = TRUE to skip missing values.

mean(mtcars$mpg, na.rm = TRUE)

Apply to bill_data$bill_area.

TipSolution 
mean_raw <- mean(bill_data$bill_area, na.rm = TRUE)
mean_raw

You should see about 729.6 mm² — a bit above the reference value of 700.

7. Trim impossible values

Step 1 — trim impossible values. Bill area cannot be negative. Remove rows where bill_area is negative (a sign error). Keep rows where bill_area is missing. Save the result as bill_clean.

Note| means “or”

Inside filter(), the vertical bar | means “or”. The condition is.na(bill_area) | bill_area >= 0 keeps a row if either the area is missing or it is zero or positive.

Writing filter(bill_area >= 0) alone silently drops the missing rows — because NA >= 0 evaluates to NA, not TRUE.

NoteHint

From ?filter: keep rows where conditions are TRUE.

mtcars |>
  filter(is.na(mpg) | mpg >= 0)

Keep missing bill areas or non-negative areas.

TipSolution 
bill_clean <- bill_data |>
  filter(is.na(bill_area) | bill_area >= 0)

50 rows — no negative values in this dataset, so nothing was removed. The one missing measurement is still there. Confirming there are no impossible values is itself a reportable result.

8. Flag distributional outliers

Step 2 — flag outliers. Using only the non-missing bill areas in bill_clean, calculate the upper fence of the 1.5 × IQR rule: Q3 + 1.5 × IQR. Save it as upper_fence and print it. Also count how many non-missing bill areas exceed that fence and save the count as n_raw_outliers. Print both values. Do not remove any rows.

NoteHint

From ?quantile: quantile(x, probs, na.rm = FALSE). From ?IQR: IQR(x, na.rm = FALSE).

q3 <- quantile(mtcars$mpg, 0.75, na.rm = TRUE)
fence <- q3 + 1.5 * IQR(mtcars$mpg, na.rm = TRUE)
n_out <- sum(mtcars$mpg > fence, na.rm = TRUE)

Use the 75th percentile and IQR of non-missing bill areas, then count values above the fence.

TipSolution 
upper_fence <- quantile(bill_clean$bill_area, 0.75, na.rm = TRUE) +
  1.5 * IQR(bill_clean$bill_area, na.rm = TRUE)
n_raw_outliers <- sum(bill_clean$bill_area > upper_fence, na.rm = TRUE)
upper_fence
n_raw_outliers

The upper fence is about 920 mm² — one penguin has a bill area above it.

9. Histogram on the raw scale

Step 3 — assess normality. Make a histogram of the non-missing bill areas in bill_clean using ggplot() and geom_histogram(). Save the plot as hist_raw and display it.

NoteHint

From ?ggplot: add a layer with +.

hist_raw <- ggplot(mtcars, aes(x = mpg)) +
  geom_histogram()
hist_raw

Map bill_area to the x-axis and assign the whole plot to hist_raw.

TipSolution 
hist_raw <- ggplot(bill_clean, aes(x = bill_area)) +
  geom_histogram()
hist_raw

The distribution is right-skewed — a long tail toward large bill areas.

10. Shapiro-Wilk on the raw scale

Run shapiro.test() on the non-missing bill areas in bill_clean. Save the result as shapiro_raw and print it.

NoteHint

Pass the bill-area vector directly.

shapiro.test(mtcars$mpg)

Use non-missing values from bill_clean$bill_area.

TipSolution 
shapiro_raw <- shapiro.test(bill_clean$bill_area)
shapiro_raw

p ≈ 0.008 — strong evidence against normality on the raw scale.

11. Square-root transform

The raw areas are not normally distributed. Apply a square-root transformation: use mutate() to add a column called sqrt_area equal to the square root of bill_area. Save the result as bill_transformed and display the first few rows with head().

NoteHint

From ?mutate: mutate(.data, ...) adds or changes columns.

mtcars |>
  mutate(sqrt_mpg = sqrt(mpg))

Add sqrt_area = sqrt(bill_area) to bill_clean.

TipSolution 
bill_transformed <- bill_clean |>
  mutate(sqrt_area = sqrt(bill_area))
head(bill_transformed)

Missing bill areas stay missing; negative rows were never present.

12. Histogram on the square-root scale

Make a histogram of sqrt_area from bill_transformed, using only non-missing values. Save the plot as hist_sqrt and display it.

NoteHint

Same pattern as the raw histogram — map the transformed column to x.

hist_sqrt <- ggplot(mtcars, aes(x = sqrt(mpg))) +
  geom_histogram()
hist_sqrt
TipSolution 
hist_sqrt <- ggplot(bill_transformed, aes(x = sqrt_area)) +
  geom_histogram()
hist_sqrt

The histogram may look a bit more symmetric, but the skew has not fully disappeared.

13. Shapiro-Wilk on the square-root scale

Run shapiro.test() on the non-missing sqrt_area values in bill_transformed. Save the result as shapiro_sqrt and print it.

NoteHint

Same function as before — pass the transformed column.

shapiro_sqrt <- shapiro.test(bill_transformed$sqrt_area)
shapiro_sqrt
TipSolution 
shapiro_sqrt <- shapiro.test(bill_transformed$sqrt_area)
shapiro_sqrt

p ≈ 0.030 — better than the raw scale, but still below 0.05. Square root did not rescue this variable.

14. Make the call on square root

Based on your histogram and Shapiro-Wilk result for the square-root data, are the transformed bill areas normally distributed enough for a parametric test? Assign sqrt_is_normal to TRUE or FALSE (no quotes, all caps).

NoteHint

Look at shapiro_sqrt$p.value. Is it above or below 0.05?

sqrt_is_normal <- FALSE
TipSolution 
sqrt_is_normal <- FALSE

p ≈ 0.030 — still below 0.05, so square root alone is not enough.

Square root is the standard transform for right-skewed counts — parasite loads, cell counts, tick burdens. Bill area is not a count. It is the product of two measurements (length × depth), and products tend to follow a log-normal distribution. Over the narrow range of real penguin bills, square root barely changes the shape — so it cannot fix the skew the way it would for worm counts.

On Coding Assignment 3, worm counts are counts, and square root is the right first transform there. Do not mechanically apply log to that assignment.

15. Log transform

Because bill area is a product, try a log transformation instead. Use mutate() to add a column called log_area equal to the natural log of bill_area. Save the updated table as bill_transformed and display the first few rows with head().

NoteHint

From ?log: log(x) returns the natural logarithm.

mtcars |>
  mutate(log_mpg = log(mpg))

Add log_area = log(bill_area) to the table that already has sqrt_area.

TipSolution 
bill_transformed <- bill_transformed |>
  mutate(log_area = log(bill_area))
head(bill_transformed)

Both transformed columns now live in the same table.

16. Histogram on the log scale

Make a histogram of log_area from bill_transformed, using only non-missing values. Save the plot as hist_log and display it.

NoteHint

Same pattern as the earlier histograms — map log_area to x.

hist_log <- ggplot(mtcars, aes(x = log(mpg))) +
  geom_histogram()
hist_log
TipSolution 
hist_log <- ggplot(bill_transformed, aes(x = log_area)) +
  geom_histogram()
hist_log

The distribution looks much more symmetric after the log transform.

17. Shapiro-Wilk on the log scale

Run shapiro.test() on the non-missing log_area values in bill_transformed. Save the result as shapiro_log and print it.

NoteHint

Same function as before — pass the log column.

shapiro_log <- shapiro.test(bill_transformed$log_area)
shapiro_log
TipSolution 
shapiro_log <- shapiro.test(bill_transformed$log_area)
shapiro_log

p ≈ 0.095 — no evidence against normality after the log transform.

18. Make the call on log

Based on your histogram and Shapiro-Wilk result for the log-transformed data, are the log bill areas normally distributed enough for a parametric test? Assign log_is_normal to TRUE or FALSE (no quotes, all caps).

NoteHint

Look at shapiro_log$p.value. Is it above or below 0.05?

log_is_normal <- TRUE
TipSolution 
log_is_normal <- TRUE

p ≈ 0.095 — no evidence against normality on the log scale.

19. Re-flag outliers after transformation

Step 4 — re-flag outliers after transformation. Using only the non-missing log_area values in bill_transformed, calculate the upper fence of the 1.5 × IQR rule on the log scale. Save it as upper_fence_log and print it. Count how many non-missing log_area values exceed that fence and save the count as n_log_outliers. Print both values. Do not remove any rows yet.

NoteHint

Apply the same Q3 + 1.5 × IQR formula to log_area instead of bill_area.

upper_fence_log <- quantile(bill_transformed$log_area, 0.75, na.rm = TRUE) +
  1.5 * IQR(bill_transformed$log_area, na.rm = TRUE)
n_log_outliers <- sum(bill_transformed$log_area > upper_fence_log, na.rm = TRUE)
TipSolution 
upper_fence_log <- quantile(bill_transformed$log_area, 0.75, na.rm = TRUE) +
  1.5 * IQR(bill_transformed$log_area, na.rm = TRUE)
n_log_outliers <- sum(bill_transformed$log_area > upper_fence_log, na.rm = TRUE)
upper_fence_log
n_log_outliers

One penguin still sits above the log-scale fence — the same Torgersen male with the large bill.

20. Decide whether to remove outliers

Based on your log-scale outlier count, decide whether to remove any rows. Assign remove_outliers to TRUE if you would filter out IQR outliers, or FALSE if you would keep all rows.

NoteHint

The one outlier is a Torgersen male with a 46.0 × 21.5 mm bill (989 mm²). That is a large but biologically plausible bill — not a data-entry error like a negative count.

remove_outliers <- FALSE
TipSolution 
remove_outliers <- FALSE

Keep the bird. Extreme but real observations belong in the dataset unless you have a biological reason to exclude them.

21. Impossible vs. extreme

One Adelie penguin had a bill area of 989 mm² (46.0 mm long × 21.5 mm deep) — above the IQR fence on every scale we checked. That value is extreme but biologically plausible. Large-billed individuals occur in wild populations. It is not a data-entry error the way a negative area would be.

The log transformation compressed the right tail of the skewed raw areas, making the distribution much more symmetric and bringing the Shapiro-Wilk test in line with the normality assumption — without throwing away real observations.

On Coding Assignment 3 you will explain this same distinction in prose for worm counts in deer mice.

Try this with a partner. One of you explains why negative counts get removed but high counts stay; the other explains why log worked here but square root did not. Swap roles.

22. Recap — verbs you practiced

Function Role
library(tidyverse) Loads dplyr, ggplot2, and related packages
filter() Keeps rows matching a condition (species == "Adelie", is.na(...) \| ... >= 0)
mutate() Adds or changes columns (bill_area, sqrt_area, log_area)
glimpse() Quick look at rows, columns, and types
mean(..., na.rm = TRUE) Average of a numeric column, skipping missing values
quantile(..., 0.75, na.rm = TRUE) 75th percentile for the IQR fence
IQR(..., na.rm = TRUE) Interquartile range for the 1.5×IQR rule
sum(x > fence, na.rm = TRUE) Counts values above an outlier fence
ggplot() + geom_histogram() Visual check of distribution shape
shapiro.test() Formal test of normality (H₀: data are normal)
sqrt() Square-root transformation (for right-skewed counts)
log() Natural-log transformation (for products and multiplicative skew)

If you can explain each row in your own words, you have covered the core of Coding Assignment 3.

23. Transfer to Coding Assignment 3

On Canvas you will repeat this pipeline with deer-mouse parasite data from RodentParasiteLoad.csv:

  1. library(tidyverse) and read.csv("RodentParasiteLoad.csv") → parasite_data
  2. mean(..., na.rm = TRUE) → mean_raw, compared to a reference mean of 12 worms
  3. filter(is.na(worm_count) | worm_count >= 0) → parasite_clean
  4. Upper fence with quantile() and IQR() → upper_fence; count outliers → n_raw_outliers
  5. Raw histogram → hist_raw; raw Shapiro-Wilk → shapiro_raw
  6. mutate(sqrt_worms = sqrt(worm_count)) → parasite_transformed
  7. Transformed histogram → hist_sqrt; transformed Shapiro-Wilk → shapiro_sqrt
  8. Assign sqrt_is_normal — for worm counts, square root should work
  9. Re-flag on the transformed scale → upper_fence_sqrt and n_sqrt_outliers
  10. Assign remove_outliers and write a short prose answer about retaining outliers

Same functions, different biological story — intestinal nematodes instead of penguin bills. Worm counts are counts, so square root is the correct transform there. Do not apply log just because this lab used it for bill area.

Keep playing

Compare two variables side by side: Gentoo flipper length (roughly symmetric visually) vs. the same measurement after every transform. Run shapiro.test() on raw, square-root, and log scales and see that none rescue this variable — the honest next step is a non-parametric test, not another transform.

All three p-values stay below 0.05. Flipper length spans only 203–231 mm, so no monotonic transform can fix a distribution that is fundamentally not normal for a parametric test.