Please enable JavaScript.
Coggle requires JavaScript to display documents.
Online Controlled experiments - Coggle Diagram
Online Controlled experiments
Fundaments
Reasons
Able to detect unexpected changes
Best Scientific way to establish
causality with high probability
Able to detect small changes
Tenets
Investment in controlled experiments
Recognition that it`s hard to assess the value of ideas
Data-driven organization with formalized OEC
Ethics Principles
Beneficence
Justice
Respect for persons
Begin
Design
Requirements
Agreement on key metrics (OEC)
that can be reliably be measured
Changes are easy to make
Users assigned to different variants
with little or no interference.
Enough users
Structure
Target population
Size of population
Randomization unit
How long
Evaluation
External validity
Different populations
Best Strategy: rerun the experiment
Over time
Novelty effect
Primacy effect
Surprising results
Replicate them
Segmentation -
Conditional Average Treatment Effects
(CATEs)
Reasons
Heterogeneous Treat. Effect, specific experiment
Metric Analysis, indep. of experiment
Good segmentations
Market/Country
Device/Platform
Time of day/day of week
User Characteristics
Attention to
Simpson's Paradox
Complementary techniques to A/B tests
Surveys
Observational studies
Human Evaluation
User Exp research
Focus groups
Logs Based Analysis
Tiered significance
to Metrics
Third order
α=0.001
Second order
α=0.01
First-order
α=0.05
Ideas for experiments
Supporting evidence
Validated metrics
Complementary metrics