Chapter 5: quasi-experiment designs that use both control groups and pretests

designs that use both control groups and pretests

the untreated control group design with dependent pretest and posttest samples

  1. untreated control group design with dependent pretest and posttest samples
  1. untreated control group design with dependent pretest and posttest samples using a double pretest
  1. untreated control group design with dependent pretest and posttest samples using switching replications
  1. untreated control group design with dependent pretest and posttest samples using reverse-treatment control group
  1. cohort control group design
  1. cohort control group design with pretest from each cohort

--- the pretest allows for exploration of the possible size and direction of that bias --- when pretest differences do exist, the possibility increases that selection will combine with other threats additively or interactively --- a selection - instrumentation threat can occur when nonequivalent groups begin at different points on the pretest. on many scales, the intervals are unequal, and change is easier to detect at some points than at others. selection-instrumentation problems are probably more acute when ..

  1. the greater the initial nonequivalence between groups
  1. the greater the pretest-posttest change
  1. the closer any group means are to one end of the scale, so that ceiling and floor effects occur

a fourth problem is selection-history. the possibility that an event occurred between pretest and posttest that affect one group more than the other

how the plausibility of threats depends partly on the observed pattern of outcomes

the plausibility of a threat is always contextually dependent on the joint characteristics of the design, on extrastudy knowledge about threats, and on the pattern of observed study results

1. both groups grow apart in the same direction this is common with maturation. standardizing of scores can make the spreading of the variables or groups disappear. if group mean differences are a result of this selection-maturation threat, then differential growth between groups should also be occurring within groups.

2. no change in the control group: when the controls do not change, the critic must explain why spontaneous growth occurred only in the treatment group. sometimes within-group analysis can shed light on between-group threats.

3. initial pretest differences favoring the treatment group that diminish over time: a scenario is when the pretest superiority of a treatment group is diminished or eliminated at posttest. selection-maturation is rare.

4. initial pretest differences favoring the control group that diminish over time experimental-control difference is greater at pretest than at posttest, but now the experimental group initially underperforms the controls. the outcome is subject to typical scaling and local history. selection-maturation is often ruled out.

5. outcomes that cross over in the direction of relationships: the trend lines cross over the means are reliably different in one direction at pretest and in the opposite direction at posttest. the outcome is particularly amenable to causal interpretation. selection-maturation is reduced. regression threat is unlikely.

ways to improve the untreated control group design with dependent pretest and posttest samples

1. using a double pretest: the same pretest is administered at two different time points, preferably with the same time delay as between the second pretest and the posttest. the double pretest allows the researcher to understand possible biases in the main treatment analysis. the double pretest permits assessment of a selection-maturation threat on the assumption that the rates between O1 and O2 will continue between O2 and O3. the assumption is testable only in an untreated group.

2. using switching replications: with switching replications, the researcher administers treatment at a later date to the group that initially served as a no-treatment control. the design can be extended to more than two groups. when it is, sometimes it is possible to assign groups at random to the particular time at which they start treatment, because by definition there must be many consecutively staggered times available if the design is to be implemented with many groups. this can help strengthen inferences, the more so when many groups at many time points are available. even without this this design is effective for the lapsed time component. the major limitations of this design follow from the fact that later instances of groups serving as controls entail either...

  1. keeping the same treatment in place but presuming it to have no long-term discontinuous effects in the same direction as the treatment later applied to the initial controls.
  1. removing the treatment from the original treatment group. this potentially sets up processes of compensatory rivalry and the like that must be thoroughly described, measured, and used in analysis. otherwise, the switching replications design is strong. only a pattern of historical changes that mimics the time sequence of the treatment introductions can serve as an alternative interpretation.

3.using a reversed-treatment control group: a NR experiment that has a pretest and two different treatments w/ a posttest. the reversed-treatment design can have a special construct validity advantage. the causal construct must be rigorously specified and manipulated to create a sensitive test in which one version of the cause affects one group one way, whereas its conceptual opposite affects another group the opposite way. interpretation of this design depends on producing two effects with opposite signs. it therefore assumes that little historical or motivational change would be taking place. when change is differential across treatments but in the same direction, results are less interpretable, because their relationship to a no-treatment control group is unknown. also, in many contexts, ethical and practical considerations prevent using a reversed-treatment. most treatments have ameliorative and prosocial goals, but a conceptually opposite treatment might be harmful.

4. direct measurement of threats to validity: these measurements allow the researcher to diagnose the possible presence of threats to validity. history threats are possible. each individual threat has to be conceptualized, validly measured, and validly analyzed, making measurement of threats difficult. if we measure correctly can help with statistical analysis.

matching through cohort controls

cohorts are partially useful as control groups if...

the crucial assumption with cohorts is that selection differences are smaller between cohorts than would be the case between noncohort comparison groups.

  1. an organization's archival records can be used for constructing and then comparing cohorts.
  1. organizations insist that a treatment be given to everybody, this precluding simultaneous controls and making possible only historical controls
  1. cohorts differ in only minor ways from their contiguous cohorts
  1. one cohort experiences a given treatment and earlier or later cohorts do not

improving cohort controls by adding pretests: consists of NR groups. one groups goes through two pretests while the other goes through one after the previous group has gone through two. the group that goes through only one pretest then goes through the treatment while the group that went through two pretests does not receive treatment. a posttest is conducted on the group that received the treatment and the one pretest. history is salient internal validity threat in this design - it can involve any event correlated with the outcome that appears only during the pretest and posttest period, even if there is a series of cohort control periods. only if a nonequivalent control group is added to the design and measured at exactly the same time points as the treatment cohorts can we hope to address history. sometimes the design can be strengthened by adding nonequivalent dependent variables if these are appropriate for the topic under discussion. another threat, testing, is possible because some comparisons involve contrasting a first testing with a second testing. a reliable difference between these two groups at posttest might be due to testing; the lack of such differences would suggest that testing effects are not a problem. finally, because causal interpretation depends on a complex pattern of outcomes in which three contrasts involve )2, a change in elevation of O2 would have crucial implications.

improving cohort designs with a nonequivalent dependent variable: if only maturation explained the results, then we would expect no difference between knowledge of letters that were taught versus those that were not

designs that combine many design elements

untreated matched controls with multiple pretests and posttests, nonequivalent dependent variables, and removed and repeated treatments

combining switching replications with a nonequivalent control group design

sometimes the researcher can introduce treatment to part of the original control group, with other controls remaining untreated over this time period. sometimes this can even be done a second time in the same experiment.

an untreated control group with a double pretest and both independent and dependent samples

the elements of design

assignment

  1. random assignment
  1. cutoff based assignment
  1. other nonrandom assignment
  1. matching and stratifying
  1. masking

measurement

  1. posttest obsevations

single posttests

nonequivalent dependent variables

multiple substantive posttests

  1. pretest observations

single pretest

retrospective pretest

proxy pretest

repeated pretests over time

pretests on independent samples

  1. moderator variable with predicted interaction
  1. measuring threats to validity

comparison groups

single nonequivalent groups

multiple nonequivalent groups

cohorts

internal versus external controls

constructed contrasts

  1. regression extrapolation contrasts
  1. normed contrasts
  1. secondary data constrats

treatment

switching replications

reversed treatments

removed treatments

repeated treatments