Please enable JavaScript.
Coggle requires JavaScript to display documents.
How can I tell whether a program actually caused a change? - Coggle Diagram
How can I tell whether a program actually caused a change?
Cause and Effect
(this is the heart of the study)
The treatment is what I change.
This is the independent variable, such as offering a mentoring and life skills program.
The outcome is what I measure.
This is the dependent variable, such as academic engagement or decision making.
A treatment group receives the program.
A control or comparison group helps me see what might have happened without it.
A pretest shows where participants started; a posttest shows where they ended.
The key question:
Did the program make the difference, or could something else explain the change?
How participants get into groups
Random assignment:
Participants are placed into groups by chance. This helps make the groups similar before the program begins and strengthens a claim about cause and effect.
Random selection:
Participants are chosen from a larger population by chance. This helps with how broadly the findings may apply.
These are different decisions.
Selection affects
who is in the study
; assignment affects
which group they join.
Without random assignment, existing differences between groups may influence the outcome.
That is selection bias.
Design Choices: what fits the real situation?
True experiment:
I provide a treatment and randomly assign participants to groups.
Pre-test and post-test design:
Measure both groups before and after the program. I can compare how much each group changed.
Post-test only design:
Measure both groups afterward. This avoids having the first test influence the second.
Covariance design:
Measure another factor that could affect the outcome, then account for it in the analysis.
Factorial design:
I study more than one treatment factor.
Main effect:
Does one factor matter overall?
Interaction effect:
Does the effect of one factor depend on another? For example, do life skills workshops work differently when youth also receive mentoring?
Hybrid designs:
I combine features to answer a particular concern.
Randomized blocks:
Compare participants within similar subgroups.
Solomon four group design:
Check whether taking a pretest changes the results.
Switched replication:
Groups receive the program in different phases, so everyone eventually participates.
Quasi experiment:
I provide a treatment but cannot randomly assign participants.
Nonequivalent groups:
Compare existing groups while paying close attention to how they differed at the start.
Regression discontinuity:
Use a set score or eligibility cutoff to decide who receives the program.
Proxy pretest:
Use an earlier record when a true pretest was not collected.
Separate pretest and posttest samples:
Compare group averages when the same people cannot be measured twice.
Nonequivalent dependent variable:
Also measure an outcome the program should not affect, to help check other explanations.
What could weaken the results?
(What else could explain the change?)
History:
Another event happened during the study.
Maturation:
Participants changed naturally over time.
Testing:
Taking the first test affected answers on the second.
Instrumentation:
The way I measured the outcome changed.
Mortality:
Participants dropped out, possibly at different rates across groups.
Regression to the mean:
An unusually high or low starting score moved closer to average later.
Selection bias:
The groups were different before the program began.
Where the study happens
Lab setting:
I have more control over outside influences, but the setting may feel less like real life.
Field setting:
I see the program in a real school or community, but it is harder to control everything else happening there.
Internal validity asks:
How confident am I that the program caused the result?
External validity asks:
Would the finding hold for other youth or settings?
Program quality matters
A clear theory connects the program to the outcome.
Why should mentoring affect decision making or engagement?
The treatment must be delivered consistently.
Otherwise, participants may be receiving different versions of the program.
A pilot test can show what needs fixing before the full study.
A manipulation check asks whether participants actually experienced the treatment as intended.
Reliable and valid measures matter.
Even a strong design cannot rescue a measure that does not capture the outcome I care about.