Tag: model-validation
Concepts
- Designing an Ablation Study
- Aleatoric vs Epistemic Uncertainty
- AUC Pitfalls Under Severe Class Imbalance
- Backtesting ML Risk Models
- Baseline Models and Naive Benchmarks
- Bayesian Optimization for Model Search
- Benchmark Design and Leaderboard Overfitting
- The Bias Statistic for Risk Model Testing
- Brier Score and Proper Scoring Rules
- Calibration Curves and Reliability Diagrams
- Champion-Challenger and Shadow Deployment
- The Chi-Square Goodness-of-Fit Test
- Clustering Stability and Bootstrap Validation
- Concept Drift Detection
- Conformal Prediction
- The Confusion Matrix and Classification Metrics
- The Corrected Paired T-Test for Cross-Validation
- Cost-Sensitive Evaluation With Asymmetric Errors
- Covariate Shift and Domain Adaptation
- Cross-Validation When Samples Are Dependent
- Data Drift vs Concept Drift
- Data Validation and Schema Checks
- Error Analysis and Slice-Based Evaluation
- Expanding Versus Rolling Training Windows
- Expected Calibration Error
- Experiment Tracking and Reproducibility
- Feature Selection Bias
- Calibrating a Forecast: Are Your Numbers Real?
- Friedman Test and Nemenyi Post-Hoc Comparison
- Holdout Set Contamination
- K-Fold Cross-Validation
- Label Shuffling and Null Models
- Learning Curves and Sample Complexity
- Lift and Gain Charts
- LIME and Local Surrogate Models
- Matthews Correlation Coefficient
- McNemar's Test for Comparing Classifiers
- Model Monitoring in Production
- Model Registry and Versioning
- Model Selection with AIC and BIC
- Choosing the Operating Point
- Partial Dependence Plots
- Estimating Live Performance Without Labels
- Permutation Feature Importance
- Precision-at-K and Top-Decile Evaluation
- Precision-Recall Curves
- Probability Calibration
- Random Forests and Out-of-Bag Error
- Regression Error Metrics
- Repeated K-Fold and the Variance of CV Estimates
- Retraining Cadence and Model Staleness
- Validating a Risk Model
- Robustness and Adversarial Evaluation
- ROC Curves and AUC
- How Much Data You Need to Detect an Improvement
- Stratified and Grouped Cross-Validation
- Subgroup Performance and Fairness Slices
- Temperature Scaling for Neural Networks
- Train-Test Decay Curves for ML Signals
- Train, Validation and Test Splits
- Utility-Based Model Evaluation
- Validating Black-Box Models for Model Risk Committees
- Validating Synthetic Market Data Realism
- White's Reality Check and the SPA Test