1.4 Model Evaluation Metrics: Precision, Recall, and F1 Score

🎯 Key AWS AI Exam Takeaways

• Precision measures the accuracy of positive calls: TP / (TP + FP). Prioritize when False Positives are costly (spam filter).
• Recall measures the proportion of actual positives caught: TP / (TP + FN). Prioritize when False Negatives are costly (cancer screening).
• F1 Score is the harmonic mean of Precision and Recall, providing an honest metric for imbalanced datasets.

A confusion matrix cross-tabulates actual ground truth against model predictions into True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN). In heavily skewed datasets where the positive class is rare, overall accuracy is dangerously misleading.

Precision evaluates the reliability of affirmative predictions: TP / (TP + FP). A high precision means that when the model sounds an alarm, it is almost certainly correct. In contrast, Recall (Sensitivity) evaluates coverage: TP / (TP + FN). High recall ensures that very few actual positive cases slip through undetected.

The F1 Score computes the harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Because the harmonic mean heavily penalizes extreme imbalances, F1 score is the industry standard benchmark for evaluating classifiers on imbalanced real-world datasets.

⚠️ Common Pearson VUE Exam Traps

• Do not trust raw accuracy on imbalanced datasets (the 'accuracy paradox').
• Remember that AUC-ROC evaluates ranking ability across ALL possible classification thresholds, not just 0.5.