Please enable JavaScript.

Coggle requires JavaScript to display documents.

A/B testing and Experiment Design - Coggle Diagram

- - - - Zoom out and Zoom in
      - If get stuck, Take a step back
      - Investigate user's behaviors
        
        By analyzing user activity logs
      - Limits: historical data only tells us past not able to predict future accurately
      - Conduct retrospective analysis by analyzing users' activity logs
      - Causal inference
    - - Focus group and surveys
        
        pain points and preferences
    - - lack of infrastructure
      - lack of impact
      - lack of traffic
      - lack of conviction
      - lack of isolation
- - - - Sample size of a test (The formula: n = 16*sigma^2/delta^2
        
        estimated Variance
        
        historical data
        
        A/A tests
        
        Validate the experiment system: make sure the assignments are correctly given and Type I error is controlled.
        
        Validate no major bias between different groups of users.
        
        Estimate the variance of certain metrics, such that we could calculate the sample size of another experiment.
        
        always run a series of a/a tests before an a/b test!
        
        simulate thousands of a/a tests and plot p-values: far from uniform? you're in trouble!
        
        common reasons behind failures: variance is calculated incorrectly (e.g., violating i.i.d. assumptions, randomization unit $\neq$ analysis unit), outliers, browser redirects, hardware differences, etc.
        
        even after a/a tests pass, it's still recommended to run a/a tests concurrently with a/b tests to ensure no regressions in the system
        
        Collect data for one week
        
        Run simulations 1000 times
        
        Goodness of fit test
        
        KS test
        
        Estimate ratio variance
        
        Follow p(1-p) maximum variance 0.25 is often used for conversion rate, click through rate.. when no prior info provided
        
        Need to monitor and check because variance can be changed over time.
        
        Rule of thumb by Power: 0.8
        
        alpha
        
        Delta: Minimum detectable effect[MDE a.k.a practical significance] (value is discussed and decided by multiple stakeholders.)
        
        The value of power is dependent on MDE
        
        It should be absolute difference (number) not the relative difference percentage
      - Dividing the sample size by the number of users in each group: Sample Size/(Randomization Units/Day)
        
        If the number is less than a week, we should run the experiment for at least seven days to capture the weekly pattern. It is typically recommended to run it for two weeks. When it comes to collecting data for a test, more is almost always better than not enough.
        
        Eg. 5% traffic, not 100% traffic to avoid buggy and terrible for all users at once.
        
        Ramping
  - - - Network effect(Spillover)
        
        lead to bias
      - Unlike social networks where the treatment effect underestimates the real benefit of a new product, in two-sided markets, the treatment effect overestimates the actual effect. :question:
      - how do we design the test to prevent the spillover between control and treatment
        
        Social network
        
        Network clusters
        
        May still have interactions
        
        Ego-cluster :question:
        
        Two sided market
        
        Geo-based (has pitfall)
        
        Sample size may not be enough
        
        Time-based randomization. (Use with care- consider time sensitive feature-short time effect)
        
        Sample size may not be enough
        
        Need monitor to detect
      - Not all experiments will be impacted by network effect
  - - - Success metrics(Goal metrics, true north star)
      - Driver metrics (more sensitive than Goal metrics)
        
        Business Goal
        
        AARRR, HEART
      - Guardrail metrics (Counter metrics)
        
        Organizational guardrails
        
        Website/App performance
        
        Latency: wait times for pages to load
        
        Error logs: number of error messages
        
        Client crashes: number of crashes per user.
        
        Business goals
        
        Revenue: revenue per user and total revenue
        
        Engagement: e.g., time spent per user, DAU, and pageviews per user.
        
        Trustworthy-related guardrails
        
        Monitor the Sample Ratio: Control and Treatment samples to be sized according to the configuration
        
        Cache hit ratio to be the same among Control and Treatment
      - Overall Evaluation Criterion (OEC)
      - Average-of-ratios vs. Ratio-of-averages
        
        click-through rate, we can use the average of ratios because it’s more robust to outliers
        
        Caveats of ratio-of-averages
        
        Less robust to outliers
        
        complicating variance calculation (need to consider using bootstrap or delta method)
    - - Randomization unit
        
        User ID or login that users use across platforms and devices
        
        Cookie
        
        Event: page view, a session (more finer granularity)
        
        Device ID
        
        IP address (not recommended)
      - How to choose
        
        Consistent user experience
        
        Variability
        
        Ethical considerations
      - Randomization unit v.s. unit of analysis
        
        Randomization unit is the same as(or coarser than) the unit analysis
    - - Demographics
      - Platforms
      - segment of interest
- - - - Conduct a statistical test to see if the two groups differ significantly.
      - Check if there’s any discrepancy upstream of the randomization point.
      - Check if the variant assignment is done correctly.
      - Check bot detection and filtering. data pipeline
      - Causes
        
        Tests with ramping up plans
        
        Running multiple tests in parallel
        
        Segmentation is based on attributes that can change
        
        Bugs in assigning pipeline
    - - Website/App performance
        
        Latency, error logs, client crashes
      - Business goals
        
        Revenue, engagement
    - - Stop and access
      - Priority concern before analyze
      - Rerun the experiment
      - How to debug SRM
  - - - Compare two proportions by Two sample t-test
        
        pt, pc, overall p
        
        Compute SE
        
        If pooling variances
        
        if not pooling variance
        
        Compute t-statistic
        
        Find p-value and compare with Alpha
        
        Old-school way(with df)
        
        SciPy
        
        Find the 95% Confidence Interval (CI)
      - Compare two means by two sample t-test
        
        Compute SE
        
        Pooled SE
        
        unpooled SE
        
        Compute t-statistics
        
        Find p-value
        
        Find the 95% Confidence Interval (CI)
      - Due to CLT, the z-test can be applied to estimate sample proportion :star:
        
        z=(p - p0)/sqrt(p0(1-p0))/n
    - - If 95% CI includes 0, it suggests no change
  - - - Unbalanced sample sizes for segments, i.e., segments that differ considerably in size
      - Post-test segmentation with unplanned comparisons between pairs or groups
    - - analyzing too many creates a much higher possibility of false positives
    - - specify the targeted population
      - stratified sampling to create balanced datasets between segments
    - - qualitative analysis.
      - a follow-up test
  - - - Internal factors
        
        Business side
        
        Launch new product/features
        
        Engineering side
        
        Latency and downtime
        
        Models and software degrade
      - External factors
        
        User's behavior changes
        
        User-learned effects
        
        Novelty effects
        
        Primacy effects
        
        Network effects
        
        Delayed experience/measurement
        
        Seasonality
        
        Competitor and others
        
        Competitors emerge
        
        Policies
    - - Long-running experiments
        
        disadvantage
        
        Survivor bias
        
        Dilution
      - Holdback Group
        
        Easier to identify potential novelty effects
        
        For bug fixes or sensitive changes, launch to entire user base
        
        Use large number of employee for internal holdback group
      - Reverse experiment
        
        To conquer the unethical problem from holdback group
      - Alternative methods
        
        Focus groups
        
        Survey
        
        Run an A/A test after the A/B test
        
        Cohort analysis
- - - - statistically significant: p<05 and lower bound in 95%CL >0
      - practically significant: new - old > Delta
    - - Scenario 1:
        
        Neither statistically and practically significant (Not launch :red_cross:)
      - Scenario 2:
        
        Statistically significant but not practically significant
        
        implementation costly (not launch :red_cross:)
        
        cost is low(launch :check:)
    - - Scenario 1: likely statistically significant , CI is too wide, CI contains 0, we cannot tell change hurt
      - Scenario 2: likely practically significant
    - - statistically significant and like practically significant
        
        CI does not have 0, there are high chances the results is practically significant
- - - - Stop the experiment earlier
      - The experiment ran as designed but there are not enough randomization units, Due to system errors or shift in demand
    - - Increase probability of false positives (Type 1 error)
      - How to fix it
        
        Group metrics
        
        Bonferroni correction
        
        Family-wise error rate
        
        FWER
        
        False discovery rate (FDR) adjustment
        
        Two steps of rules of thumb
        
        Divide the metrics into three groups: set different p-value according to the impact level predicted
      - senarios
        
        More than 2 metrics
        
        More than two treatment groups
        
        a segment of population
        
        Multiple iterations
        
        Multiple testing in parallel
    - - Lack of statistical power each segment
      - Prevent: Before running test keep balanced and same ration of segmentation of each group for both controlled and treatment group
      - After running test, can do some qualitative test
    - - results in highly skewed results
- - - - interrupted time series
      - interleaved experiments
      - regression discontinuity design
      - propensity score matching
      - difference in differences
- - - - Bootstrapping :star:
        
        To invoke the CLT
      - Running alternative tests
      - Gathering more data
    - - Z and t-tests