Threats to Validity

Confounds

extraneous variables that systematically vary or influence both the independent and dependent variables

  • may have a different confound in your experimental and control groups.
  • a major threat to internal validity
  • where do confounds come from? the experimenter, participants (individual differences, demand characteristics and participant reactance), and the effect of time.

third variable problem

  • the best way to account for systematic confounds is usually to manipulate the independent variable.
  • pseudo/false experiments are most susceptible
  • importance of control groups
  • importance of isolating the variable of interest.

control groups

  • you want to manipulate only the variable of interest between groups/conditions
  • the challenge is to keep everything else constant
    • almost impossible to isolate and remove all other variables
    • need to try to keep all other variables outside of your control constant and matched in experimental and control group

Internal Validity Threats
from the Experimenter

  • experimenter bias is a confound that undermines the strength of a causal claim.
  • the bias of the experimenter may influence the way a dependent variable is scored.
  • the experimenter may behave in a way that influences the participants and confounds the results of the experiment
  • not always intentional
    • previous knowledge and ideas can create tunnel vision in the experimenter
  • double-blind procedures neither the experimenter nor the participant are aware of the conditions
  • prevents both unintentional and intentional bias when interpreting data.

Internal Validity Threats
from the Participant

  • the way a participant can influence the validity of the results.
  • the individual differences that are systematic can interfere with the causal relationship you are investigating.

participant expectancy

  • participants identify the purpose of the study > behave in a certain way as a result

individual differences

  • differences in characteristics between all individual participants, including animals.
  • when there is a systematic issue with individual differences there is a threat to internal validity.
  • need to try to control for these differences, but they are impossible to completely remove.

click to edit

participant reactance bias

  • when a person modifies their behaviour because they are participating in a research study.
  • a feature of the study suggests what the purpose of the study is.
  • this changes your ability to measure the behaviour of interest.

participant reactivity

  • good subject role: tries to produce the response the researcher wants.
  • negative subject role: tries to produce the response contrary to what the researcher wants.
  • apprehensive subject role: overly concerned about their performance.
  • faithful subject role: follows instructions and ignores suspicions

demand characteristics

  • a feature of the study suggests what the purpose of the study is > the participant changes their behaviour in response.
  • surveys and scales are prone to this > difficult to hide the purpose of the study in questionnaire research

solutions

  • unobtrusive observation: hidden video camera
  • indirect measures: you can't just ask people things, you have to go about it indirectly > waiting for them to notice.
  • deception/confederates: dummy participants

Time-Related
Confounds

maturation: the effect of time on participants

  • short term: mood, tiredness, hunger, boredom
  • deal with this by: counterbalancing the order of tests, controlling for the time of day, designing experiments of a reasonable length, including breaks.
  • long-term: age, education, wealth
  • difficult to control, though only important for long longitudinal studies.
    • random assignment and sampling helps reduce.

artifacts: something that is ever-present in all groups being tested and stays constant

  • reduces external validity
  • prevents you from generalising your results.

mere measurement effect: being aware that something is observing or measuring your behaviour may change the way you behave > hawthorne factory
-important for external validity as it undermines the ability to generalise our results to the wider population.

  • similar to participant reactance, except that it affects all subjects in the experiment

history effects: the effect of a period of time may make an entire sample biased

  • the data is influenced by the moment in time
  • can't generalise these findings to a wider population or different contexts

selection bias: participants who have a biased interest in the topic of research or the outcome of the study volunteer

  • important to use a random sample
    • doesn't eliminate all problems but reduces the likelihood of systematic biases.
  • compulsory poll- census
    • reduces sampling bias
    • produces results and data that are more widely acceptable
    • less susceptible to biased groups
    • still susceptible to demand characteristics

non-response bias: problem for experiments that involve voluntary sign-ups or surveys

  • people do not response when they're not interested > lose a large sample of the population
  • undermines external validity > limited population means that the results cannot be generalised to a wider population.