Chapter 7: Decision analytic thinking- Good modeling

Concepts

Techniques

Topic overview:

Estimating cost/benefits

Calculating Exp. profit

Various evaluation metrics

Creating measurement baselines

Expected value as evaluation of framework

Evaluation of comparative measurement baselines

End purpose of results

  1. Outlining potential surrogates for data that is not available for any reason
  1. General evaluation methods for data problems and general business problems alike

3 main points:

  1. Clearly defining end goals for data mining and data operations

Key analytic framework: Expected value

Evaluation, baseline performance, and implications for investments in data


Evaluating classifiers

Problems with unbalneced classes

Problems with unequal costs and benefits

Confusion matrix

Generalizing beyond classification

Accuracy and its problems

Using expected value to frame classifier use

Using expected value to frame classifier evaluation

what to watch for:

Expected value

Expected value decomposes data analytic thinking

Example: expected value of pr(x) * vr + [1-pr(x)] + vrn

Do not double count (neg, loss, pos, gain)

remain consistent with focus EV (gain/loss)