Please enable JavaScript.
Coggle requires JavaScript to display documents.
Product case interview (PCIP), hi - Coggle Diagram
Product case interview (PCIP)
Measuring success
Step1: Understand the goal
Understand the functionality of the feature
What
How
Who
Understand the goal
Make an educated guess and double-check with the interviewer
Or ask the interviewer
Step2: Follow the user journey or funnel
User journey
How do users see this feature
How do navigate
How to interact with it
Experience funnel
Only a certain percentage of users will progress though each step of the funnel
Step3: Define the metrics
Define 1-2 success metrics
For A/B tests define 1 guardrial metric
For two-sided/three-sided marketplace define more metrics if the interviewer requires
for ratio metrics, show the numerator and the denominator
Two mistakes :no_entry:
Using high-level metrics
not sensitive enough to capture the success or failure of feature/product changes
DAU, which can be impacted by many features and initiatives
capture a lot of unintended noise in the data
Disregarding the timeframe of the metric
Typical A/B tests run only for a few weeks
The longer a time frame is, the fewer pieces of data a metric will contain
14 data points of daily metrics
2 data points of weekly metrics
Metrics to measure
User engagement
Conversion rates
Retention
Launch or not
Step 1-3: Same as measuring success
Step4: Design Experiments
Definition of control and treatment groups
Randomization unit
Experiment run-time
Common pitfalls and fixes
Long-term monitoring
Determining the minimal detectable effect
Calculating sample size
Step5: Interpret results
Launch if
Success metrics are all practically and statistically significant
No violation of assumptions
No negative change of guardrail metrics
Do not launch if
Violation of assumptions (How :question:)
Negative change of guardrail metrics
Discuss trade-offs
:star:
Unify both positive and negative impact to one metric
Segmentation of positive and negative on user groups and features
Discuss whether the overall result is desirable
Sanity checks to make sure the data is reliable
Diagnosing a problem(
Example
)
Step1: Clarify the problem and the change of the metric
How things are defined
How metrics are calculated
Are they ratio metrics? numerator or denominator change?
How much were the changes
relative and absolute terms?
Are we comparing this change to last week or on a year-over-year basis?
Importance?
Step 2: Clarify the timeframe for the change
Sudden change
Anything happened internally within the company?
Any data generation processes or bugs with instrumentation and logging change that is not reflective of user behavior?
new feature launches, bug fixed, new marketing campaign?
Any external events or obvious market/traffic shifts?
Competitors launch new products, pandemic or recession
gradual change
Need more investigation
At what granularity? over a day, a week, a moth?
Natural change
seasonality? day of the week, holiday, weather?
Step 3: Generic across multiple products OR specific to one product or feature
Have other related products/features experienced the same change?
Narrow the scope and context of the problem
Features are cannibalized
Walk through the product funnel to analyze
The UI, then ranking models...
Step4: Segment users or product features to validate each factor
Is the change associated with any user/product segments?
User segmentation:
Geographic location
Age group
Device type
Product/feature usage tier
segment users by their demographic and behavioral features) but also segment by other attributes of a product.
Language
Product segmentation
Specific feature enabled/disabled
attributes of a product.
by content
Prioritize and narrow down
Step 5:Decompose the metric
Study change of each component
Active users = existing users + new users + resurrected user - churned users
Cohort analysis
Narrow down the cause of the change
Step6: Summarize
Classify Each factor
Root cause
Contributing factor
Correlated result
Unrelated factor
To make sure it is healthy
Improving a product
Step1: Understand the goal
Who
the ideal use case for this product
Understand the functionality of the feature
Step 2: Explain your approach
Be open and communicative about your thought process with your interviewer.
Step 3: Identify product improvement opportunities
Go over the user journey and identify friction points.
Requires data that can reveal key user needs.
Analyze current users' behaviors to identify the needs of current users and the different needs of different user segments.
Step 4: Identify a solution and prioritization
Judge solutions by overall cost-effectiveness
Select ideas that have the largest impact
Step5: Define metrics
Define 1 to 2 success metrics
For A/B tests
Define 1 guardrail metric
Step 6: Summarize your overall approach
Outline the goal
Solutions
How to prioritize them.
Making strategic decisions
Step1:Understand the goal
Understand the functionality of the feature
What
How
Who
Understand the goal
Ask the interviewer
Or make an educated guess and double-check with the interviewer
Step2: Strategic analysis
Use the 5Cs framework, think of
Customers
Company
Mission
Care about sustainable growth
Collaborators
Competitors
Context
Step 3: Summary
Product specific questions
Sample questions
We get many ideas from our product teams, and we cannot A/B test each idea due to resource constraints. As a data scientist, how would you select which ideas to invest in?
How do you evaluate the impact of fake news on Facebook?
How do you determine the optimal ratio between company posts and individual posts for LinkedIn Feed?
Step1: Understand the company’s business model and challenges
What are the goals of the company?
What stage has been reached for these goals?
What are the unique challenges being faced by the company currently?
Step 2: Utilize existing data
(To learn)
What can be learned from historical data?
Can anything be learned in terms of user behaviors and preferences?
Step 3: Break down the problem into smaller pieces
User segmentation
Geographic location
Age group
Device type
Product/feature usage tier
new vs existing
Product segmentation
Specific feature enabled/disabled
Step 4: Combine quantitative and qualitative methods
Quantitative methods
existing data
external data
A/B testing (Note: A/B testing isn’t suitable for all use cases)
Qualitative methods
User research
Focus group testing
Surveys
Ask clarifying questions :star:
How the metric is defined
If the question involves a change in metrics, it’s important to clarify how much the change is.
How the product/feature/program works (if the question includes a new feature/product/program). If it’s a feature that existed for a long time, eg. Facebook News Feed, it’s better to be familiar with it and do the research before the interview.
Who is the target audience of a new product/feature?
The goal of the product or a feature. We recommend you make an educated guess instead of asking directly because often the goal is clear.
...Am I on the right track?
How does that sound
Estimation
Tips
Clarify
Be coachable
Remain Pragmatic
Mention tradeoffs
Timebox yourself
hi