Regression Artifacts — Regression to the Mean

Overview

This exercise demonstrates regression to the mean — a statistical phenomenon that occurs whenever a group is selected on the basis of an imperfectly measured variable. The simulation is designed to convince you that regression effects are real, predictable, and entirely the product of imperfect measurement. No program, no treatment, no change in true ability is needed to produce what can look like a meaningful gain or decline.

The key insight is this: when you select people who scored low on one test, you have inadvertently selected a group that contains an unusually large proportion of people who had a bad day (i.e., whose random error on that test was negative). On a second test, measured independently, those same people are unlikely to have equally bad days. So on average they appear to improve — not because anything happened to them, but because the extreme errors that put them in the low group are unlikely to repeat.

Step 1 — Generate the Data

Create two test scores, X and Y, from the same true score T. The true score has mean 50 and standard deviation 10; the errors for each test have mean 0 and standard deviation 5.

Step 2 — Examine the Full Sample

Look at descriptive statistics and the bivariate distribution for the full sample of 500. Compute the correlation between X and Y — this value will be used later in the regression formula.

Step 3 — Select the Below-Average Group (X < 50)

Suppose we want to give a special remedial program to all students who scored below the mean of 50 on the X test. Select this group and compare their mean on X versus their mean on Y.

The mean of Y for this group will be higher than their mean of X. It looks like these students improved between the two tests — even though no program was given and no time passed. This apparent gain is entirely a regression artifact.

To understand why, examine the error sign distributions. Among the full sample, negative and positive errors are approximately equally common. But in the group selected for having a low X score, negative X errors are overrepresented — many of those students scored low on X partly because of bad luck on test day.

Step 4 — Select the Above-Average Group (X > 50)

Now look at the change for students who scored above average on X. Their mean on Y will be lower — they appear to have regressed downward. If you didn't know the data were simulated, you might conclude that high scorers were harmed.

Step 5 — Extreme Low Scorers (X < 40)

Select a very extreme low-scoring group (below 40) to see regression to the mean more dramatically. The more extreme the selection, the larger the apparent gain.

The Regression-to-the-Mean Formula

We can predict the magnitude of the regression effect using the correlation between the two tests. The expected percent of regression toward the overall mean is: