RMB - Theory

Research Approaches

  1. Experimental
  • Assess causality
  • Compare groups (between/within) - random assignment
  • Manipulate IV, measure DV
  • Control extraneous variables
  • Gold standard - not always possible
    Pros: causality/directionality, high internal validity
    Cons: low external validity, can't always randomise, hard/time-consuming

Designs

Within-groups / repeated measures

  • Test same group 2+ times (all do each treatment) - multiple scores / participant
  • BUT could anything else be responsible? occurred naturally?
  • Pros: less participants, eliminates individual diff, more powerful
  • Cons: can't use if carryover effect, time-related factors, may be too long
  • Counterbalancing: admin in diff order - removes time effects (fatigue, practice) - complete or partial
  1. Quasi-experimental
  • Address causality (not infer) - control groups, multiple measures
  • Has confounding variable (usually pre-existing groups)
  • Pros: addresses causality, sometimes only option if experimental impossible/unethical
  • Cons: non-equivalent groups, limited control of extraneous variables, assignment bias
  1. Non-experimental
  • Examine relationship (no causality)
  • No manipulation
  • Pre-existing groups
  • Pros: quicker, higher external validity
  • Cons: no causality, low internal validity
  1. Correlational
  • Describe relationship
  • Pros: non-intrusive, high external (no manipulation), good for prelim, good if unethical/impossible to manipulate
  • Cons: no causality, low internal, 3rd variable problem + directionality problem
  1. Descriptive
  • Describe current state

Validity

Internal: conclusion valid?
(Uncontaminated by extraneous influences)

External: conclusions generalisable?

For causality

  1. IV precedes DV
  2. Relationship between IV & DV
  3. Rule out other causes

Open Science

  • Collaboration, pre-registration

Between-groups / independent groups

  • Compare different groups (e.g. treatment vs control)
  • Participants do 1 treatment - 1 score / participant
  • BUT groups might not have been equal to begin
  • Pros: always option, no time effects
  • Cons: more participants, large individual diff. have effect, assignment bias

Mixed: between-groups + within-groups

  • Randomise to control vs treatment
  • Measure both groups before & after

Threats: questions/doubts about study

Extraneous variable: extra variable (not measured/interested in)

Confounding variable: extraneous variable that has effect on measured variables (alternative explanation)

Environmental: time of testing, diff experimenters

Assignment bias: group participants may vary => randomisation

Time

  • History: events during study
  • Maturation: natural change in participants
  • Instrumentation: tech issue / researcher skill
  • Testing effects: fatigue, practice; carryover effects => use counterbalancing
  • Regression to mean: extreme 1st score tends to be less extreme on 2nd testing => more time points
  • Attrition

Participants: selection bias, volunteer bias, using uni students

Balancing internal/external validity: increasing one often decreases other

  • More control => more internal, less external
  • More real world => more external, less internal

Features of study: reactivity/demand effect - setting, experimenter, multiple treatment interference, experimenter characteristics

Measurements: sensitisation, specific measures used, timing of measurement

Controlling extraneous variables

  • Hold constant, but limits generalisability
  • Balance/match, but can't control for many
  • Random assignment, but hard for small groups

Developmental

Cross-sectional: groups tested once at same time; between subjects

  • Pros: time efficient, no selective attrition, no practice effects
  • Cons: individ. changes not assessed, cohort effects

Longitudinal: test participants > once; within subjects

  • Pros: no cohort effects, assess individual change
  • Cons: attrition bias/selective attrition, practice effects, reduced generalisability, biased sample

Cross-sectional longitudinal: diff people (each time) tested once at diff points in time

Longitudinal-sequential: 2+ groups, followed longitudinally

  • Pros: detect cohort effects, cross-sectional & longitudinal comparisons
  • Cons: expensive

From:

  • Sample to population
  • Study to another study
  • Study to real world

Sampling

Scales

Qual Research

Populations vs Samples

  • Population: entire set of interest
    • Target: of interest + hard to reach all
    • Accessible: of interest + can reach
  • Sample: set selected from pop, to represent pop, participate in study

Representative samples

  • Representative sample: sample with same characteristics as population
    • Important for generalising
  • Biased sample: sample that has diff characteristics to population, because of
  • Sampling error: diff b/n sample & pop. due to people selected
    • Chance
    • Selection / sampling bias: sampling procedure favours some people over others,
      e.g. non-response bias (people who respond are diff)
    • Small sample size
  • Non-sampling / measurement error: error due to how observations are made, e.g. poor instruments / procedures, interviewer effect, respondent effect, knowing study purpose, induced bias
  • Sampling: process of selecting individuals

Probability Methods

  1. Entire pop. known (size, individuals)
  2. Specified probability of selection
  3. Unbiased / random: equal chance of selection
    -> impractical/impossible, rarely used

Non-Probability Methods

  1. Entire pop. not known (size, individuals)
  2. Odds of selection not known
  3. Biased method of selection - common sense, ease
    -> greater risk of biased sample

Codesign

  • Participants involved in decisions that affect them
  • Best practice for Indigenous research

Simple random

  • Every individual has equal chance of selection
    • With replacement: independent
    • Without replacement: not selected > once
  1. List all members
  2. Randomly select
  • Can be biased because of chance

Systematic

  • Same as simple random to select 1st participant
  • Then select every nth participant

Stratified random

  • Select equal-sized, random samples from subgroups
  • -> All subgroups equally represented, for comparing subgroups

Proportionate stratified random

  • Select random samples, so sample proportions match pop. proportions
  • -> Sample composition matches pop. composition

Cluster

  • Select random pre-existing groups of people
  • -> Quick, easy, BUT less independent

Convenience

  • aka accidental, haphazard sampling
  • Use easy to get, available & willing
  • Pros: easy, cheap, timely
  • Cons: little control over representativeness, likely biased

Quota

  • Fill quotas for subgroups
  • Can match population %

Purposeful

  • Select info-rich cases, knowledgeable, available
  • For in-depth study
  • Often used in qual research
  • Not trying to generalise

Ethics

  • Pregnant women / foetuses
  • Children / young people
  • Dependent / unequal relationships
  • Dependent on medical care / unable to consent
  • Cog impairment, intellectual disability, mental illness
  • Illegal activities
  • Other countries
  • ATSI people

Definitions

  • Construct: abstract concept
  • Variable: concrete representation
  • Operational definitions: specified methods/procedures for measuring variables - repeatable, objective
    • No single variable or op. def. completely captures construct -> use multiple
    • Allows re-measurement

Generating operational definitions

  • Define construct -> generate operational definition
  • Nomological net: define in terms of other constructs (based on theory & common sense)
  • Interviews & theme analysis: collect qual data -> themes -> def
  • Research: lit review, theory -> def
  • Direct observation: note behaviours -> def
  • Expert judgement: ask expert -> def

Constructing self-report measures

  • Pros: direct, easy, cheap
  • Cons: easy to distort
  • Types
    • Open-ended Q: flexible, access true thoughts BUT hard to compare/summarise, limited by willingness/ability of participants
    • Restricted Q (limited options): easy to analyse/summarise, quant info BUT less flexible
    • Rating-scale Q (numeric scale): easy to analyse/summarise (numeric interval), easy to understand/answer BUT response set
      • Likert: continuum, numbers with equal spacing, agree/disagree, average?/sum
      • Semantic differential: pairs of bipolar
      • 5-10 categories: can't discriminate more

Assessing scales

  • Does construct exist?
  • Useful?
  • Valid?
  • Reliable?
  • Items appropriate?

Administering surveys/scales

  • Interview:
  • Mail:
  • Phone:
  • Internet: economical, efficient, flexible BUT not representative, non-response bias, hard to control respondents
  • Non-response bias: individuals who complete survey not representative
  • Interviewer bias: interview influences responses (e.g. tone, rephrasing Q)

Validity & Reliability

  • Need reliability to be valid
  • Do NOT need validity to be reliable - can be reliable BUT not valid

Validity

  • Whether measures what it claims to
  1. Face: looks like it measures what it claims to (superficial)
  2. Concurrent: related to established measure
  3. Predictive: predicts behaviour (according to theory)
  4. Construct: works as theory / past research suggests - grows gradually
  5. Convergent: 2 diff methods for same construct -> related
  6. Divergent: same method for 2 different construct -> not related

Approaches

  • Phenomenology: meaning
  • Ethnography: organisation / culture, immersive
  • Grounded theory: dev theory from real world data, iterative data collection / analysis

Methods

  • Interviews
    • Structured, semi-structured, unstructured
    • Face-to-face, phone; one-on-one, group (6-12)
  • Observation: participant, structured, 'episodes'
  • Document analysis

Analysis

  • Themes: common thread in data
  • Prelim: emerging themes, ID gaps, ongoing analysis
  • Thematic analysis: stop when saturated, coding

Principles

  • Purposiveness: purpose -> RQ -> method -> data -> analysis
  • Congruence: fit / consistency between problem -> RQ -> method -> data -> analysis in way of thinking
  • RQ or data may demand qual approach

Why / when?

  • Little known or complex
  • Discover new theory
  • Understand in detail / how participants understand
  • Accuracy, not reproducibility

Reliability

  • Consistency (with same people & conditions)
  • Large error -> low reliability:
    1. Observer error: measurer
    2. Environmental changes
    3. Participant changes
  • Types
    • Test-retest: for successive measurements - time sampling
    • Inter-rater: for simultaneous measurements
    • Internal consistency: compares groups of items (e.g. split-half, Cronbach's alpha) - content sampling

Items

  • Sensitivity: discriminates high / low
  • Simple: brief, clear, unidimensional, no double-barrelled, no double -ves
  • No item bias: emotional, personal, inaccessible
  • Response set: tendency to give same response to most Q -> use alternate responses (+ve & -ve), but -ve places extra demand
  • Infrequency check: checks random responses