Larsen Chapters 2-7 Mindmap (Chapter 7: Success, Risk, and Continuation…
Larsen Chapters 2-7 Mindmap
Machine Learning Lifecycle
Not uncommon to have to cycle back when progress is made
IE Get to the "Interpret and Communicate" stage, have to take another look at "define project objectives"
Careful adherence to some rules can minimize the risk of this though
Criteria For AutoML Excellence
Accuracy - Most important. Need validation procedures etc
Productivity - Perpetually living with a voice in the back of your head telling you to think of a new and better algorithm
Ease of Use - Should minimize the amount of machine learning knowledge necessary to effectively use the platform
Understanding and Learning - Should improve an analysts understanding of the problem and problem context
Resource Availability - Should be compatible and integratabtle with other tools in the business ecosystem
Process Transparency - Important in order to build trust for the process and in order to be able to more easily compare models
Generalizable Across Contexts - Should work for all target types, data sizes, time perspectives. Need to be able to use for cross-sectional as well as longitudinal data types
Recommend Actions - Mostly for context-specific AutoML; should be able to interpret and fully understand decisions to be able
Chapter 3: Specify Business Problem
Starting With a Business Problem
This is the best place to start as it gives you a good question to answer and address. In order to best address it, it should be well specified. This way we can evaluate options better
Is the statement presented in the language of business?
Does the project statement specify actions that should result from the project?
How could solving this problem impact the bottom line?
Chapter 4: Acquire Subject Matter Expertise
Without subject matter expertise, complete insights cannot be provided on a business problem
Need deep experience. Helps with identification of potential roadblocks and obstacles in the process
Need it in order to set realistic expectations for model performance
Chapter 5: Decide on Unit of Analysis
What Is a Unit of Analysis?
The what, who, where, when of our analysis
Often times there are many different ways to discern UoAs and many options as far as which to use
How to Determine It
Sometimes it is far from obvious. In such cases the subject matter expert should weigh in
Chapter 6: Define Prediction Target
What is a Prediction Target?
The behavior of a "thing" we need to know about the future
Why is it important?
Without a target there is no way for humans or machines to learn what associations drive an outcome
Helpful when trying to make machine learning conduct regression
Chapter 7: Success, Risk, and Continuation
Identifying Success Criteria
Factors to Consider
Who will use the model?
Is management on board with the project?
Can the model drivers be visualized?
How much value can the model produce?
Difficult to calculate by nature. To get at the risks you need to be creative and often play devils advocate
Deciding Whether to Continue
Weigh risk and reward
Evaluate worthiness of getting additional data
Consider pilot projects. Should either succeed or fail quickly in order to minimize risk