This is the second exercise in the analysis of nonequivalent group
designs. In the first exercise you learned how to simulate data
for a pretest - posttest two-group design. You put in an initial
difference between the two groups (i.e., a selection threat) and
also put in a treatment effect. You then analyzed the data with
an Analysis of Covariance (ANCOVA) model and, if all went well,
you found that the estimate of program effect was biased (i.e.,
the true effect was not within a 95% confidence interval of the
obtained estimate).
In this exercise, you will create data exactly as you did in the
last exercise. This time however, you will adjust the data for
unreliability on the pretest before conducting the ANCOVA. Because
unreliability or measurement error on the pretest causes the bias
when using ANCOVA, this correction should result in an unbiased
estimate of the program effect.
By now you should be fairly familiar with what most of these MINITAB
commands are doing (remember, you should consult the MINITAB Handbook
or use the HELP command if you want information about a command)
and so the initial command will be presented without explanation.
Get into MINITAB and you should see the MINITAB prompt (which
looks like this MTB>). Now you are ready to enter the following
commands:
MTB> Random 500 C1;
SUBC> Normal 50 5.
MTB> Random 500 C2;
SUBC> Normal 0 5.
MTB> Random 500 C3;
SUBC> Normal 0 5.
MTB> Add C1 C2 C4.
MTB> Add C1 C3 C5.
MTB> Name C1 ='true' C2 ='x error' C3 ='y error' C4 ='pretest' C5 ='posttest'
MTB> Set C6
DATA> 1:500
DATA> End
MTB> Code (1:250) 0 C6 C6
MTB> Code (251:500) 1 C6 C6
MTB> Name C6 = 'Group'
MTB> Let C4 = C4 + (5 * C6)
MTB> Let C5 = C5 + (15 * C6)
You have constructed the data exactly as in Part I. Notice that
the data commands from the previous exercise have been condensed
here. If you don't understand why you used these commands, go
back and read the instructions for Part I again. Be sure that
you understand that you now have data for two groups, one of which
received a program or treatment (i.e., those with a '1' for C6)
and the other a comparison group (i.e., with a '0' for C6). The
groups differ on the pretest by an average of five points, with
the program group being advantaged. On the posttest, the groups
differ by fifteen points on the average. It is assumed that five
of this fifteen point difference is due to the initial difference
between groups and that the program had a ten point average effect.
You should verify this by examining the group means:
MTB> Table C6;
SUBC> means C4 C5.
You should also look at histograms and at the bivariate distribution:
MTB> Histogram C4
MTB> Histogram C5
MTB> Plot C5 * C4
MTB> Plot C5 * C4;
SUBC> Symbol C6.
Recall that the ANCOVA yields biased estimates of effect because
there is measurement error on the pretest. You added in the measurement
error when you added C2 (x-error) into the pretest score. In
fact, you added in about as much error as true score because the
standard deviations used in all three Random/Normal commands are
the same (and equal to 5). In order to adjust the pretest scores
for measurement error or unreliability, we need to have an estimate
of how much variance or deviation in the pretest is due to error
(we know that it is about half because we set it up that way).
Recall that reliability is defined as the ratio of true score
variance to total score variance and that we estimate reliability
using a correlation. Traditionally, we can use either a split-half
reliability (the correlation between two randomly-selected subsets
of the same test) or a test-retest correlation. In these simulations
we will correct the pretest scores using a test-retest correlation.
Since the test-retest correlation may differ between groups it
is often advisable to calculate this correlation for the program
and comparison groups separately. Before we can do this we have
to separate the data for the two groups and put it in separate
columns:
MTB> Copy C6 C4 C5 C20-C22;
SUBC> use C6 = 0.
MTB> Copy C6 C4 C5 C30-C32;
SUBC> use C6 = 1.
MTB> Name C21 = 'contpre' C22 = 'contpost' C31= 'progpre' C32 = 'progpost'
The first statement copies all cases having a '0' in C6 (i.e.,
all the comparison group cases) and their pretest and posttest
scores (C4 and C5) and puts these scores in C20 through C22.
It is important for you to notice that the order in which you
copy the variables (i.e., C6, C4, C5) is the order in which they
are put into the other columns. Therefore, C20 should consist
entirely of '0' values. The next statement copies all the program
group cases and puts them in C30 through C32. We are not interested
in C20 or C30 because these consist of the '0' and '1' values
so we don't name them. We name C21 'contpre' to stand for control
pretest, C31 'progpre' to stand for program group pretest, and
so on. Now, we are ready to estimate the test- retest reliabilities
for each group. These are simply the correlations between the
pretest and posttest for each group:
MTB> Correlation C21 C22 M1
MTB> Correlation C31 C32 M2
Each correlation is stored in a 2x2 matrix variable. In order
for us to use these correlations later, we have to copy them from
the M matrix into a K-variable constant. We can do this with
the following commands:
MTB> copy M1 C41 C42
MTB> copy C41 K1;
SUBC> use 2.
MTB> copy M2 C43 C44
MTB> copy C43 K2;
SUBC> use 2.
Now, we will adjust the pretest scores for the comparison group.
All you need to do is the following formula:
MTB> Let C23 = aver (C21) + (K1 * (C21 - aver (C21)))
Be sure to type this statement exactly as is. If you make a mistake,
just re-enter the command. Now, let's adjust the scores for the
program group. Use the following formula:
MTB> Let C33 = aver (C31) + (K2 * (C31 - aver (C31)))
You should recognize that for each group you are taking each person's
pretest score, subtracting out the group pretest average, multiplying
this difference by the within-group pre-post correlation (the
K-variable), and then adding the group pretest mean back in.
For instance, the formula for this reliability correction for
the control group is:
We should probably name these new adjusted pretests:
MTB> Name C23='contadj' C33='progadj'
Now, compare the pretest means for each group before and after
making the adjustment:
MTB> Describe C21 C22 C23 C31 C32 C33
What do you notice about the difference between unadjusted and
adjusted scores? Did the mean change because of the adjustment
(compare the means for C21 and C23 for instance)? Did the standard
deviations? Remember that about half of the standard deviation
of the original scores was due to error. How much have we reduced
the standard deviations by adjusting? How big is the pre-post
correlation for each group? Is the size of the correlation related
to the standard deviations in any way?
Now, we want to combine the pretest scores for the two groups
back into one set of scores (i.e., one column of data). To do
this we use the Stack command:
MTB> Stack (C23) (C33) (C7).
Which means 'stack the scores in C23 on top of the scores in
C33 and put these into C7'. Now let's name the adjusted pretest
scores:
MTB> Name C7 = 'adjpre'
Now take a look at the means and standard deviations of the original
pretest, adjusted pretest and posttest by group:
MTB> Table C6;
SUBC> means C4 C7 C5;
SUBC> stdev C4 C7 C5.
It should be clear to you that the adjustment for measurement
error on the pretest did not affect the means, but did affect
the standard deviations. You are now ready to conduct the analysis.
First, do the ANCOVA on the original pretest scores (C4) just
like in Part I. You should see that the estimate of effect is
biased. Then, conduct the analysis using the adjusted pretest
scores (C7). This time you should see that the estimate of effect
is much closer to the ten points that you originally put in.
First, the biased analysis:
MTB> Regress C5 2 C4 C6
Remember that the estimate of the program effect is the COEFFICIENT
in the table which is on the C6 GROUP line. Is it near a value
of ten? You can construct a 95% confidence interval for this
estimate using the ST.DEV. OF COEF. which is on the same line
as the estimate. To construct this interval, first multiply the
standard deviation of the coefficient by two (remember from statistics
that the 95% confidence interval is plus or minus approximately
two standard error units). To get the lower limit of the interval
subtract this value from the coefficient in the table, to get
the upper limit add them. Does the true program effect (10 points)
fall within the bounds of this interval. For most of you it will
not (just by chance, it may for a few of you. Event if it does
fall within the interval it will probably not be near the center
of the interval). Now, conduct the same ANCOVA analysis this
time substituting the adjusted pretest scores (C7) for the original
ones:
MTB> Regress C5 2 C7 C6
Again, look at the coefficient in the table that is on the same
line as the C6 GROUP variable. Is this value closer to the true
program effect of 10 points than it was for the previous analysis?
It should be (again, for some of you this may not be true just
by chance alone). Once again, construct the 95% confidence interval
for the coefficient using the standard deviation of the coefficient
given in the table. You should see that this time the true program
effect of 10 points falls within the interval.
You have just conducted a reliability-corrected Analysis of Covariance.
In practice, we know that test-retest and split-half reliability
estimates will often differ, sometimes considerably. Because
of this, we will often conduct two analyses like the one above-
once using the test-retest correlations and once using the split-half
correlations. Although this will give us two estimates of the
program effect, we expect that one is a conservative adjustment
while the other is a liberal one. We can therefore be fairly
confident that the true program effect probably lies somewhere
between the two estimated effects.
At this point, you should stop and consider the steps that were
involved in conducting this analysis. You might want to try one
or more of the following variations:
Simulation Home Page