Please enable JavaScript.
Coggle requires JavaScript to display documents.
DS101: Data Science Interview Overview - Coggle Diagram
DS101: Data Science Interview Overview
Interview softskill Tips
Product Sense Interview
Specific context for the product not just follow framework
Always clarify questions, understand context before offering suggestions
Three ideas are good enough
Ask for some time to structure my answer
:red_flag: Too many ideas, using framework blindly, not be able to defend yourself
Consider potential issues and pitfalls
Ask why before addressing the problem
Strategic thinking
Company looking for
Structure
Comprehensiveness
Feasibility
Why the STAR method does not work
Goal
Improve user experience?
help promote businesses?
Generate advertisement revenue?
user growth, user engagement, increase revenue, acquisition, retention, activation
Impact
Challenges
Finding
Business
Business process
Business requirements
Data sources
Business model
E-Commerce
Saas
Journey / Conversion Funnel
Subscription
Stickiness, engagement, retention
Metrics
Acquistion, churn, DAU, DAU/MAU, average time spent per user per day, Revenue(conversion rate, revenue per customer, LTV)
Mobile application
Two-sided marketplace
Social media
Search engine
Behavior Interview
Conflict and time management
Leadership
Responsibility for the team
Thrive in ambiguity
Questions
About the company
Why do you want to join? What do you value most
About you
Tell me about yourself
Impactful project
Emphasize the impact
Make it more conversational
The situational quesiton
Tell a good story
Ask questions
Presentation
Focus on your impact.
Use your best stuff.
List the limitations.
Think through the technical details.
Remember that behavior does matter.
Projects
Goal
Impact
Challenges
Share one interesting finding
Define success metrics, obtain data, size of data large scale, processing data, data quality, model training and deployment
3 tips
zoom out and think about the big picture:why it is important project
Engage the interviewer
Remove useless details
Avoid taking non-stop and talking details according to level of interest
Preparing Interview Guideline
Focus on one type of job
Learning deeply
Doing things, learning from mistakes: Practice, Practice, Practice
Learn two items every day, roll and review
SQL Interview Basics and Tips
Easy(5'), Medium(10'), Hard(15')
Catch edge cases
OEC(Overall Evaluation Criterion)
Random unit without bias
Correlation does not imply causality
A/B testing:Running and Analyzing Experiments - A practical End-to-End example:
Setting up example
Funnel
User journey
Hypothesis Testing: Establishing statistical significance
Design Experiment
Randomization unit, population, size, how long, primacy and novelty effects
User?, Target population in the experiment, determine the sample size
n = 16*sigma^2/(delta^2) (based on alpha=0.05, Beta/power=0.80)
alpha = 0.01 so n = 18.93 sigma^2/delta^2
Sigma is from historical data (estimated variance)
Choosing Randomization Units
User ID, Cookie, Event(page view and session), Device ID
Ensure Consistent user experience
User visible change(User ID)
Non user visible change(Event/Session)
The randomization unit should be coarser than unit of analysis
Segment difference
Simpson's paradox
We need domain knowledge to find which is better(aggregated or disaggregated)
Why simpson paradox happened? unbalanced dataset, maybe another variable affect the result in subgroups
(a trend appears in different groups of data but disappears or reverses when these groups are combined). The reasons for Simpson’s paradox happening could be: (1) The setup of your experiment is incorrect; (2) The change affects the new user and experienced users differently.
Running experiment and getting data
Set up instruments and data pipelines to collect data
Interpreting the results
From results to decisions
Launch decision
Consider, metric trade-offs, cost of launching
Test a new ranking algorithm
the business goal of the problem
2.User Journey (Funnel)
Success metrics: measurable, attributable, sensitive, timely
Hypothesis Testing
Alpha, power, minimum detectable effect(MDE)
Delta is difference between control and treatment (Minimum detectable effect)
Validity checks
Bias - Instrumentation effect , Checks: Guardrail Metrics(Latency time)
Bias - External Factors, Checks: Holidays, competition, economic disruptions(e.g. covid)
Bias - Selection bias, Check: A/A test
Run A/A test if there is no historical data
Bias - Sample Ratio Mismatch, Checks: Chi-square Goodness of fit Test
Bias - Novelty effect, Check segment by new and old visitors
Three types of questions
Diagnose problem
Clarify the scenarios and metrics
How to define ETA/...? By hour or by day, aggregated by day?
Check outliers, is there extreme value, investigate the extreme value
Was the data collected correctly?
Technique issues?
Analyze the temporal factor
Change suddenly ? Tech prediction algorithm, data collection process
Progressively? Look at historical trend, weekly pattern? Compare with Ground truth
Segment users by different dimension by region and platform
Shall I summarize my approach or go ahead do further analysis
Measure success
Launch or not - Make decisions (experiment)
Organizational Metrics
Goal metrics
Success metrics/True North Metrics, singe , small set, ultimate success, take long time to materialize
Drive metrics
Shorter-term, fast-moving, more sensitive, HEART framework, AARRR, user funnels
Aligned with goal, actionable
Senstive
Resistant to gaming
Guardrail Metrics
protect and access trustworthiness, most sensitive
Examples: Ad revenue and app performance.
Two types
Business
Internal Validity
App's performance, bugs, loading time, # of errors
Sample ratio mismatch
Evaluating metrics
Lifetime value(LTV)
Select metrics
Qualitative(survey)
Quantitative (Log)
ML questions
What is Machine learning, explain to a 7 year old boy
What is overfitting, how do you deal with it
2-3 sentences, give an example, provide solutions
Imbalance data set
random forest classifier
3 evaluation metrics
L1 v.s. L2 regularization
hyperparameters
What is XGbooster? Why using it? how to evaluate?
Model validation
Cross-validation
estimate how accurately a model will accomplish in practice.
limit problems like overfitting
Applied ML problem
Entire workflow, getting and cleaning the data, building and evaluating models
Metric Interview
Two success metrics and one guardrail metric
True north metric
Statstics interview
Probability
Distribution
Normal (search time) mean, median, mode…
Exponential(time spent) mean, median, mode..
Normal distribution: 66-95-99.7 rule
CLT
rule of thumb is 30 samples
Average follows CLT
Sum also follows CLT
Test a normal distribution
Binomial
Number of clicks follows binomial distribution
Click-through rate follows normalized binomial distribution(Bernoulli distribution)
Long tail distribution
Geometric distribution
Average customer lifetime
Hypothesis Testing
Explain p value and CI to non-technical audience
Z-test and t-test
Bernoulli distribution (Click through probability, click through rate and conversion rate)
Is it a proportion or not?
Binomial test
The relationship between CI and H0:H1/Ha
inferential techniques which use a sample to either estimate a population parameter or test the strength and validity of a hypothesis.
In other words, if the null hypothesized value falls within the confidence interval, then the p-value is always going to be larger than 5%. Conversely, if the null hypothesized value falls outside of our confidence interval then the p-value is going to be less than 5%.
When 0 is included in our confidence interval this means we are likely seeing that there is no difference between our sample and the population parameter.
significant level
(It depends on the population standard deviation), p-value can use a web application
Regression
Assumptions
LINE
Linear
Use residual plots
Independent
Use residual plots
Seasonal, time serial
residuals are Normal
Nice to have
Central limit theorem
Use Q-Q plots to check
Bow- or S-shaped is not normal
Equal variance(Homoscedasticity)
Residual plots
Explain terminology
Where and when to use
Definition
When value changes what does it mean with larger or smaller value
Application
For a non-technical audience
Use examples
Avoid to introduce more technical terms
Sampling and Simulation problems
Bias coin from Fair coin
Fair coin from Bias coin
Why sampling, what is sampling, how to sample
Resampling
Why we need resampling:when normality is not held, we are not able to collect more data.
Bootstrap (with replacement)
1000 to 10000(Rule of thumb) times
Permutation test(without replacement)
Repeat 1000 times in combined set
Bayes Theorem
Total probability
Conditional probability
Central tendency: Mean, Median, Mode
Variability(Dispersion)
Variance: How spread of the data in a distribution
The sum of squared deviations from the mean divided by n – 1 where n
is the number of data values
MSE
(SD)Standard deviation: Square root of varience
L2 norm
MAE(Mean absolute deviation)
L1 norm
Percentile(Quantile)
Interquartile range (IQR)
The difference between the 75th percentile and the 25th percentile
correlation
The strength of a linearity between two variables
Pearson correlation= covariance/standard deviation
How to measure correlation in the presence of outliers? IQR method, Scaling
Correlation coefficient
Correlation matrix
Scatterplot
P value: A Conditional probability, given null hypothesis is true
P value is a measure of the strength of the evidence against the null hypothesis
Outliers
Visualize
Drop or cap
Standardization
This reduces the effect of the outliers
https://www.rapidinsight.com/blog/handle-outliers/
Sample
Standard error
Mean
The larger the sample, the standard error is smaller, the sample mean will approach closer to the population mean. The smaller the standard error, the more representative the sample will be of the overall population.
Tests
t-Tests
Multiple testing
ANOVA
F-tests
Chi-Square Test
Multi-Arm Bandit Algorithm
Pairwise tests
Variance Tests
Hypothesis Testing
Two Sample Test of proportions
Bernoullie population
Used pooled SE
Two Sample test of means
Is result practically significant
if have similar variance use pooled variance
if not have similar variance use un-pooled variance
pooled standard error
Resume
Impact, Concise and show by measure and metrics
Projects from Google
Geo retention insights
ML project in resume
Formulating a metric
Identify an action to measure
Count
Time
Value
Chose the unit of analysis
Per session
Per user
Per page
Per time
Choose a statistical function
Average
Total
Count
Median
Framework
AARRRg
HEART