Experimental Methods 8.

Topics of Lecture

  • Understand Humans and their Behavior
  • to find Causal (Cause-Effect) Relations
  • Variation as a Prerequisite to Measure Differences
  • What is Experimental Design
  • Validity in an Experiment

Understand Behavior

Uncertainty
makes us Uncomfortable

we want Control and
Predictability

Research Design

3 main
Design Approaches:

Statistical Control

  • Descriptive Design (Describe)
  • Correlation Design (Describe, Relation and Predict)

Experimental Control

Experimental Design

Science

Goals:

  • Describe
  • Predict
  • find Causes
  • Explain and Understand
  • Explain Causes
  • Explain Effects
  • find Cause-Effect Relations

Relations

Causal Relation
(Cause-Effect)

a Variable Directly or Indirectly
Influences Another

Correlational
Relation

Changes in a Variable are Only
ASSOCIATED w Changes in Another

Cause

what Produces
a Modification, a Result or a Consequence

Effect

what has been Produced

the Condition
to talk bt Causal Relation

we have Control
over the Involved Variables

Problems w
Causality

Different Causal Relations

X is Sufficient for Y

  • if X then Y
  • there can be Y wout X, if Y can be also Caused by Z

X is Necessary for Y

  • there is No Y wout X
  • there can be X wout Y

X Contributes to Y

X is a Contributing Cause to Y

  • X Can be part of a Causal Chain that Can lead to Y

Abolsute Causality
is Problematic

DIfferent Degrees of Probability

Probabilistic Causality

Sum up

image

Variation as a Prerequisite to
Measure Differences

Classical Test Theory
(aka: True Score Theory)

Spearman, early '900

Goal

Increase Reliability

Each Observation (X) is a Sum of:
True Score (T) +
a Measurement Error (e)
image

True Score (T)

Measurement Error (e)

Systematic E.

Ex.:

  • Social Desirability, Demand Characteristics,
    Unclear (/Bad) content, Experimenter Bias,
    Lying, Motivation

Affects Validity

we are Not Sure that
we have Measured what we Intended to Measure

Ex. if we want to Measure Alcohol Intake during Pregnancy
and everybody Lie bc they know it's Wrong

have we rellay measured Alcohol Intake
during Pregnancy or Smth else?

to Control
Systematic Errors

Very Difficult
History of Researchers Headache :)

Unsystematic
(Random) E.

Ex.:

  • daily shape, Situational conditions (noise in the room...),
    error by test administrators, memory

Affects Reliability

you do Not get the
Precise Scores you wanted

Does Not Influence Data
in a Particular Direction

to Control
Unsystematic Error

  • just Replicate the Test again and again
  • or Replicate it w Other Groups
  • See Reliability Testing

Reliability Testing #

  • allows to Discover Unsystematic Errors,
  • while Systematic E. are Camouflaged
    as a part of the True Score

can be Calculated Out: image

Value from 0 to 1

1 = all Variation
is due to True Scores

we Measure Only the Same Smth,
which equals to True Score (since we Ignore Systematic Errors)

0 = all Variation
is due to Errors (Unsystematic)

we do Not Measure the Same Smth. at all,
we Measure Only Unsystematic/Random Errors

Systematic and Unsystematic Variation
in Experiments

Casual Unsystematic Variance

Systematic Variation

we have Control on Variation

we Intervene in the Experiment
by Manipulating a Variable

the Variation is due to the Manipulation

the Variation is Not Natual,
is Artificial

it may also be Due Confounding Variables

but we try to Avoid them

all the Variation Not due to
our Manipulation

we hope is due to
Unsystematic Variation

and not due Confounding Variable

What is Experimental Design

Experiment

Definition

Formal

Introducing an Event (Indipendent Var.) in a Controlled Environment,
to Measure the Effect of the Variable on Other Variables

Scientific Method used to
Test Hypotheses of Cause and Effcet

Control Group

Randomized Participants

Each Participant has the
same Chance to be Selected

No Bias in Selection

click to edit

Types

Between-Subject

(At least) 1 Experimental Group
and 1 Control Group

Within Subjects

the Same Group is Measured Before and After the Manipulation

N=1 Design

1 Individual is followed Over Time

Quasi-Experimental

Same as Experimental Design,
but Lacks Randomized Assignment of Participants

can Talk bt Causal Relations

when a Participant that receives the Manipulation
can be Switched w Another Participant
and we get the Same Results

Things we should
have Control over:

  • Sample: Representative
  • Control and Experimental Groups
  • Indipendent Variable
  • Dependent Variable
  • External Conditions: control over External
    Variables that may Possibly have an Effect on Y

Validity in Experiments

Test Validity

  • Content Validity
  • Criterion Validity (Concurrent, Predictive)
  • Construct Validity (Convergent, Divergent)

Inferential Validity

  • Internal Validity
  • External Validity

image image

Statistical Validity

External
Validity

if the Findings are Generalizable
(to the Real World)

External to the Controlled Environment

if we have the Statistical Basis to draw
our Conclusions

Concerns:

  • Samping
  • Analyis Method
  • Number Precision
  • Representativity
  • etc.

Internal
Validity

Threats

Between-Subjects Exp.

Selection Bias

Diffusion of
Indipendent Variable

Definition

there are Systematic Diffrences
btw the Experimental and Control Groups
Before the Manipulation

Selective Drop-Out Rate

Within Subjects Exp.

Carry Over
Effect

Previous Measurements
have a Transissible Effect

ex. Irreversibility that Carries Over to
following Measurements

Repeated Testing - Learning

Fatiguing

Tired, do Not Perform as Well

Habituation

Repeated Stimulus receives
a Weakened Response

Sensitization (more Psychological)

  • Used to Stimuli
  • Expecting Stimuli

Constrasts

Adaptation

Physical Adaptation

History

Contrasts Betwee different Conditions may make us more Attentive to Conditions that come After

so that these Later Conditions actually get More Attention than they should have

ex. Tolerance to medicaments

Changes in the Circumstances
that are not due to Changes in X

ex. Changes in Society,
in your Family, in You,
Death in Familiy etc.

Changes in the Features
of the Measuring Instruments

Statistical Regression

Scores, especially Extreme ones,
have a Tendency to move toward the Average
(in the next Measurement)

Maturing

Natural Changes in the Person
during the Time Interval

Threats

Observation Effects /
Reactivity

  • Experimenter Bias
  • Hawthorne Effect
  • etc.

how to Tackle them

Invert the Order of Conditions
for Each Participant

Change the Order in which you
Introduce Indipendent Variables

ex. you have 4 situations A, B, C, D.
You Change their Order

Pre-Test Design

have a Pre-Test Post-Test Group

Test Before Manipulation,
Control Baseline Behavior

Solomon 4 Groups Design

Pre-Test (Pr), Post-Test (Po):

  1. Pr ... X ... Po
  2. ... X ... Po
  3. Pr ... ... Po
  4. ... ... ... Po
  • All have a Post-Test
  • Only one has both Pr, X and Po
  • one has only Pr
  • the last has only X

Formal

  • whether Researchers have Drawn the Correct Conclusions on the Conditions of Cause-Effect in the Experiment
  • Inferences have Internal Validity if a Causal Relation btw 2 Variables is Demonstrated
  • the extent to which a Causal Conclusion based on a study is Warranted, which is determined by the degree to which a study Minimizes Systematic Error (or 'Bias')
  1. the Cause Precedes the Effect
  2. the Cause and the Effect are Related (Covariation)
  3. there are No Plausible Alternative Explanations for the Observed Covariation (Nonspuriousness)

Personal

  • when we Control all Extraneous Variables and the Only Variable Influencing the Results is the One being Manipulated (Indipendent V.)
    and Not other Extraneous Variables
  • the Degree to which the Experiment supports Clear Causal Conclusions

the Effect of the Manipulation is Carried Over
to the Control Group

Ex. In the study of ppl. running more on the tapis roulant when seeing their progress on the diplay

is whether the increase of performance (running) was only due to the display (indipendent variable) and not other extranoeus variables

Ex. in the study of the tapis roulant

whether the findings (the relation between the ind. variable of seeing the progress on a display and the dep. var. of increased running) apply to other contexts, like the impotance of measurement in systematic goal setting