Chapter 2 - How to keep score (H. Segment, Cohort, A/B Testing, and…
Chapter 2 - How to keep score
H. Segment, Cohort, A/B Testing, and Multivariate Analysis
A segment is simply a group that shares some common characteristic.
How to define segments:
According to a range of technical and demographic information, then compare one segment to another.
Segment A: visitors using the Firefox browser.
Segment B: visitors using Safari browser.
Definition: "Longitudinal studies"
Data is collected along the natural lifespan of a customer group such the user in first month and the user in the fifth month.
Definition: cross-sectional studies
Compare one attribute of a subject's experience, such as, link color and assuming everything is equal
G. Moving Target P.21
in the early stage, you don't know how to define the success, it's like you're drawing a line in the sand-not carving it in stone.
What you do:
Adjusting your goal and how you define your key metrics is acceptable.
In the reality:
There is a gulf between what you assume and what users actually do. For example, you might think that people will pay your multiplayer game, only to discover that they're suing you as a photo upload service.
HighScore House Defines an "Active Users"
Know your customer:
A combination research with quantitive data (to monitor the behaviours) and qualitative data(to discover the reasons) is necessary to adjust your key metrics.
C. Vanity v.s. Real Metrics
Ask “What will I do differently based on this information?”
Total signups / Total active users / Percent of users who are active
Number of hits
Number of page views.
Number of visits.
Number of unique visitors
Number of followers/friends/likes.
Time on site/number of pages.
Number of downloads
D. Exploratory v.s. Reporting Metrics
F. Correlated v.s. Causal Metrics
B. Qualitative v.s. Quantitative Metrics
If quantitative data answers “what” and “how much,” qualitative data answers “why.”
You need to ask specific questions without leading potential customers or skewing their answers.
I. A Lean Analytics Cycle
E. Leading v.v Lagging Metrics
A leading metric is to predict the future.
A lagging metric 讓你知道現有的狀況
A. What Makes a Good Metrics?
Use ratio or rate
Easier to take action
Good for comparing factors that are opposite but you should consider
Change the way you behave
Check whether the actual results are convergin on your business
Optimize! Learn sth new? Need to try sth new?