Please enable JavaScript.

Coggle requires JavaScript to display documents.

A/B Testing - Coggle Diagram

- - - - Steps:
        
        Define high-level concept of metrics
        
        Click-through-probability: #users who click / #users who visit #
        
        Define the details of the metrics (what do you mean by active users?)
        
        Def.1: for each <time interval>: # cookies click / #cookies
        
        Def.2: Pageviews w/ click in <time interval> / pageviews
        
        Summarize that every measure into single metrics (e.g. mean)
        
        Categories:
        
        Total metrics (count, sum)
        
        users who visited the page
        
        Distribution metrics (mean, median, percentiles)
        
        Mean age of users who completed the course
        
        Probabilities and rates
        
        0 or 1 in each case (click or no click)
        
        Ratios
    - - OEC -- Overall Evaluation Criteria
  - - - ROBUSTNESS: A/A test and check different metrics. If the metric variates across video -- its not robust. 95% and 99% percentiles are not robust
      - SENSITIVITY: A/B test for decreasing your metric. If you increase the video quality, the latency should decrease
- - - - 2 types of outcomes
      - Independent events
      - Identical distribution (p same for all)
- - - - Compare result to what you expect (sanity check)
      - Estimate variance and calculate confidence
      - directly estimate confidence interval
  - - - Pooled standard error (for multiple samples)
    - - Margin of Error (m)
        
        Confidence interval #
    - - H0: p_new = p_old or p_new-p_old = 0
        H1: p_new != p_old
      - distance (d) = p_new - p_old
        
        if d more or less z*SE => reject H0
      - Always compute the metric's variability
        
        Firstly, define variability analytically (compute)
        
        It is necessary to check for "invariance" (sanity checks) #
        
        After the results, make a sign-test (computed p-value)
        
        Multiple metrics
        
        In case of multiple metrics, you should use the Bonferoni correction
        
        If you use the multiple metrics in one test, you increase the probability of False Positive result (because every metric has the 5% (100%-95%) probability of False Positive)
        
        1 more item...
        
        Secondly, define it empirically (based on A/A tests)
    - - alpha = P(reject H0 | H0 True)
      - beta = P(not reject H0 | H0 False)
        
        1 - beta = sensitivity (80% usually)
      - Smaller sample
        
        alpha low
        
        beta high
        
        Increased practical significance (d_min)
- - - - User ID
      - Anonymous ID (Cookie)
      - Event
      - Device ID (rare)
      - IP adress (rare)
    - - User consistency (if user occured is control/test group, she should stick in it)
      - Extra ethical review -- if we collect new data e.g. cookies + e-mail, it will become sensitive
      - Variability
        
        When we compute variability analytically, we make a big assumption of the data independence
        
        If we use event-based diversion, our assumption of independence is valid and analytical and empirical variabilities will be close
        
        When unit of analysis = unit of diversion => variability will be lower
        
        If we use ID- or cookie-based diversion, our independence assumption is not so valid, so analytically computed variability will me much hire than empirical
        
        Unit of Diversion -- denominator in your metric