Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 8?: Randomized experiments: rationale, designs, and conditions…
Chapter 8?: Randomized experiments: rationale, designs, and conditions conducive to doing them
the theory of random assignment
reduces the plausibility of other effects interfering with the results
it can yield unbiased estimates of the average treatment effect
What Is Random Assignment
it is achieved by assigning units to conditions based only on chance
random assignment
is not
random sampling
random sampling ensures that answers from the sample approximate what we would have gotten had we asked everyone in the population
random assignment
, by contrast, facilitates causal inference by making samples randomly similar to each other, whereas
random sampling
makes a sample similar to a population
why randomization works
it ensures that alternative causes are not confounded with a unit's treatment condition
it reduces the plausibility of threats to validity by distributing them randomly over conditions
it equates groups on the expected value of all variables at pretest, measured or not
it allows the researcher to know and model the selection process correctly
it allows computation of a valid estimate of error variance that is also orthogonal to treatment
random assignment and threats to internal validity
the randomized experiment does so by distributing these threats randomly over conditions. so treatment units will tend to have the same average characteristics as those not receiving treatment. the only systematic difference between conditions is treatments
random assignment only makes it likely that confounds are spread across the groups of study, treatment and control groups
selection bias is unlikely because for it to occur that would mean a systematic way of assigning people to groups would have to have been used.
10 situations conducive to randomized experiments
when demand outstrips supply
when demand for service outstrips supply, randomization can be a credible rationale for distributing service fairly
when an innovation cannot be delivered to all units at once
often it is physically or financially impossible to introduce an innovation simultaneously to all units
when experimental units can be temporally isolated
although we typically think of randomly assigning people, schools, communities, or cities to conditions, we can also randomly assign times to conditions
when experimental units are spatially separated or interunit communication is low
when units are geographically separated and have minimal contact with one another, or when they can be made this way, those units can be randomly assigned
when change is mandated and solutions are acknowledged to be unknown
sometimes, all concerned parties agree that an undesirable situation needs changing, but it is not clear which changes we should make despite passionate advocacy of certain alternatives by interested parties
when a tie can be broken or ambiguity about need can be resolved
assignment of people to conditions on the basis of need or merit is often more compelling rule to program managers, staff, and recipients than is randomization
when some persons express no preference among alternatives
even if ethics or public relations require that people be allowed to choose which options they will receive, persons who express no preference from among the options can be assigned by chance
when you can create your own organization
random assignment is an accepted part of the organizational culture of laboratory experimentation, but most field experiments are conducted in organizational cultures in which randomization is mostly foreign
when you have control over the experimental units
being able to establish one's own organization or randomization clearinghouse is rare
when lotteries are expected
lotteries are sometimes used as a socially accepted means of distributing resources
equating groups on expectation
it is much like a deck of cards, we are not all going to get a good hand every draw, but we expect to get the same kinds of hands over the course of the game. this is similar with people in groups, we expect that they will be equal over the course of treatment, but to look at two individuals and expect to them to be the same due to random assignment is absurd.
the longer the study the more likely the means from the treatment and control groups are likely to average out
additional statistical explanations of how random assignment works
specification error
: an incorrect specification of the model presumed to give rise to the data
the error is usually the difference between the study and the population averages or means
random assignment and units of randomization
a "
unit
" is simply an opportunity to apply or withhold the treatment
kinds of units: units can be assigned to anything from people to schools or neighborhoods to even objects in natural science experiements
higher order units are larger things like pediatrician offices spread out across a state and lower order units are the people that go to the pediatrician office
studies that use higher order units frequently have fewer such units available to randomize
when more than one, but still fewer, higher order units are assigned to conditions, randomization may result in very different
means
,
variances
, and
sample sizes
across conditions
given the same number of individual units, power is almost always lower in designs with higher order units than in those with individual units; and special power analysis must be used
a way to help with error and variance is to assign or switch treatments half way through the experiment so that way the control groups and the treatment group go through both types of treatment or lack of treatment
the limited research of random assignment
if the units do not make sense it makes the research almost useless
some deigns used with random assignment
1. the basic design
: requires at least 2 conditions, random assignment of units to conditions, and posttest assessment of units
2. two variants on the basic design
: both groups are treatment groups, but one is a new treatment and the other is the gold standard to measure what we are trying to measure. the goal is to see if the new treatment is any more or less effective than the gold standard
risk to this design due to lack of pretest
those who dropped out of the study were different from those who remained
if those who dropped out of one condition were different from those who dropped out of the other conditions
the pretest-posttest control group design
adding pretest to the randomization of groups is highly recommended
alternative-treatment design with pretest
the addition of pretests is also recommended when different substantive treatments are compared.
if not difference occurs than the research can measure the pretest and posttest to scores to reevaluate his/her findings
multiple treatments and controls with pretest
the randomization experiment with pretests can involve a control group and multiple treatment groups
factorial designs
these designs use two or more independent variables (called factors), each with at least two levels.
they often require fewer units
the allow testing combinations of treatments more easily
they allow testing interactions
nested and crossed designs
crossed design
: each level of each factor is exposed to (crossed with) all levels of all other factors
nested design
: some levels of one factors are not exposed to all levels of the other factors
longitudinal design
add multiple observations taken before, during, and after treatment, the number and timing of which are determined by the hypotheses under study
allow examination of how effects change over time, allow use of growth curve models of individual differences in response to treatment, and are frequently more powerful than designs with fewer observations over time, especially if five or more waves of measurement are used
multiple posttests are common in longitudinal studies
attrition is very high in longitudinal studies
crossover design
participants go through 2 postttests. the treatment and control groups go through the firs phase and receive posttests and then the groups are switched, the experiment done again with another posttest taking place after all of that is done
when random assignment is not feasible or desirable
we may need answers quickly and randomization takes time and resources
randomization provides precise answers, but the organization may not need the answers or stats to be exact
randomization experiments can rarely be designed to answer certain kinds of questions
before conducting an experiment, a good deal of preliminary conceptual or empirical work must be done