Generating Data — True Score Theory

Overview

In this simulation you will generate two imaginary test scores for 500 individuals using a measurement model called True Score Theory. Imagine that you have administered two achievement tests to 500 school children. True Score Theory assumes that every observed test score is composed of two parts — true ability and random error:

Here O is the observed score, T is the latent true ability, and e is random error. In real research you only observe O; you never see T or e directly. By simulating the data yourself, however, you know the true values — which makes it possible to evaluate how well statistical methods recover them.

We will create two tests, X and Y, each measuring the same true ability with independent errors:

Both tests share the same true score T, so any difference between a child's X and Y scores must come entirely from measurement error.

Step 1 — Generate True Scores and Errors

The first block generates 500 true scores from a normal distribution with mean 0 and standard deviation 3, and two sets of random errors with mean 0 and standard deviation 1. The true score has a larger standard deviation than the errors, which means each test will reflect more true ability than noise — a situation of reasonably high reliability.

Check that the means are close to zero and that sd(T) is about three times larger than sd(eX) and sd(eY). Small deviations from the specified values are expected because the data are randomly generated.

Step 2 — Construct Observed Test Scores

Add the true score to each error term to produce the two observed tests.

The means of X and Y should both be close to zero (the mean of T), and their standard deviations should be slightly larger than sd(T) alone because each observed score contains added error variance.

Step 3 — Examine the Distributions

Plot histograms of each test score. Both should be approximately bell-shaped because they are the sum of two normal variables.

Step 4 — Bivariate Distribution

Plot X against Y to see their joint distribution. Because the two tests share the same true score, you expect a positive correlation. The cloud of points will be elongated diagonally; the width of the cloud reflects the amount of measurement error relative to the true score variance.

Step 5 — Regression of Y on X

Fit a simple linear regression of Y on X and examine the slope coefficient. If the two tests measured true ability perfectly (no error), the slope would be exactly 1.0 because X and Y would be identical. With measurement error the slope will be less than 1.0 — a phenomenon called attenuation. The more error in X, the flatter the regression slope.

The R-squared from this regression estimates the reliability of the tests (roughly r_XX'). Compare it to the ratio var(T) / var(X) — they should be close.

Step 6 — Residuals Plot

Plot the residuals against the fitted values. If the regression model is appropriate, you expect a random horizontal scatter with no obvious patterns.