Chapter 10: Experimental Research
Experimental research, often considered to be the “gold standard” in research designs, is one of the most rigorous of all research designs. In this design, one or more independent variables are manipulated by the researcher (as treatments), subjects are randomly assigned to different treatment levels (random assignment), and the results of the treatments on outcomes (dependent variables) are observed.
Experimental research is best suited for explanatory research (rather than for descriptive or exploratory research), where the goal of the study is to examine cause-effect relationships.
Laboratory experiments, conducted in laboratory (artificial) settings, tend to be high in internal validity, but this comes at the cost of low external validity (generalizability), because the artificial (laboratory) setting in which the study is conducted may not reflect the real world. Field experiments, conducted in field settings such as in a real organization, and high in both internal and external validity
Basic Concepts:
Treatment and control groups. In experimental research, some subjects are administered one or more experimental stimulus called a treatment (the treatment group) while other subjects are not given such a stimulus (the control group).
The treatment may be considered successful if subjects in the treatment group rate more favorably on outcome variables than control group subjects. Multiple levels of experimental stimulus may be administered, in which case, there may be more than one treatment group.
Treatment manipulation. Treatments are the unique feature of experimental research that sets this design apart from all other research methods. Treatment manipulation helps control for the “cause” in cause-effect relationships.
Any measurements conducted before the treatment is administered are called pretest measures, while those conducted after the treatment are posttest measures.
Random selection and assignment. Random selection is the process of randomly drawing a sample from a population or a sampling frame. This approach is typically employed in survey research, and assures that each unit in the population has a positive chance of being selected into the sample. Mortality threat refers to the possibility that subjects may be dropping out of the study at differential rates between the treatment and control groups due to a systematic reason, such that the dropouts were mostly students who scored low on the pretest.
Threats to internal validity. Although experimental designs are considered more rigorous than other research methods in terms of the internal validity of their inferences (by virtue of their ability to control causes through treatment manipulation), they are not immune to internal validity threats.
History threat is the possibility that the observed effects (dependent variables) are
caused by extraneous or historical events rather than by the experimental treatment.
Maturation threat refers to the possibility that observed effects are caused by natural maturation of subjects. Testing threat is a threat in pre-post designs where subjects’ posttest responses are conditioned by their pretest responses.
Regression threat, also called a regression to the mean, refers to the statistical tendency of a group’s overall performance on a measure during a posttest to regress toward the mean of that measure rather than in the anticipated direction.
Instrumentation threat, which also occurs in pre-post designs, refers to the possibility that the difference between pretest and posttest scores is not due to the remedial math program, but due to changes in the administered test, such as the posttest having a higher or lower degree of difficulty than the pretest.
Two-Group Experimental Designs
The simplest true experimental designs are two group designs involving one treatment group and one control group, and are ideally suited for testing the effects of a single independent variable that can be manipulated as a treatment.
click to edit
Pretest-posttest control group design. In this design, subjects are randomly assigned to treatment and control groups, subjected to an initial (pretest) measurement of the dependent variables of interest, the treatment group is administered a treatment.
Posttest-only control group design. This design is a simpler version of the pretest posttest design where pretest measurements are omitted. Covariance designs. Sometimes, measures of dependent variables may be influenced
by extraneous variables called covariates.
Factorial Design: Two-group designs are inadequate if your research requires manipulation of two or
more independent variables (treatments).
The most basic factorial design is a 2 x 2 factorial design, which consists of two
treatments, each with two levels (such as high/low or present/absent).
Hybrid Experimental Designs
Hybrid designs are those that are formed by combining features of more established
designs.
click to edit
Randomized block design. This is a variation of the posttest-only or pretest-posttest control group design where the subject population can be grouped into relatively homogeneous subgroups (called blocks) within which the experiment is replicated.
Solomon four-group design. In this design, the sample is divided into two treatment groups and two control groups. One treatment group and one control group receive the pretest, and the other two groups do not.
Switched replication design. This is a two-group design implemented in two phases with three waves of measurement.
Quasi-Experimental Designs: are almost identical to true experimental designs, but lacking one key ingredient: random assignment.
Many true experimental designs can be converted to quasi-experimental designs by omitting random assignment. For instance, the quasi-equivalent version of pretest-posttest control group design is called nonequivalent groups design
Likewise, the quasi-experimental version of switched replication design is called non-equivalent
switched replication design.
Regression-discontinuity (RD) design. This is a non-equivalent pretest-posttest design where subjects are assigned to treatment or control group based on a cutoff score on a preprogram measure.
Proxy pretest design. This design, looks very similar to the standard NEGD (pretest-posttest) design, with one critical difference: the pretest score is collected after the treatment is administered.
Separate pretest-posttest samples design. This design is useful if it is not possible to collect pretest and posttest data from the same subjects for some reason.
Nonequivalent dependent variable (NEDV) design. This is a single-group pre-post quasi-experimental design with two outcome measures, where one measure is theoretically expected to be influenced by the treatment and the other measure is not.