Proper Scoring for High-Stakes Classification Pipelines

This evidence synthesis examines evaluation incentives in probabilistic forecasts and threat labels. It argues that the appropriate object of evaluation is the chain from public signals and historical records to probability, label, action, and feedback into future data, not a model score in isolation. The review brings together the assigned studies with established work on uncertainty, robustness, provenance, and governance. Across these literatures, a common problem emerges: overconfident predictions can shift markets or defenses and thereby invalidate the data-generating process assumed by the model. The proposed framework separates evidence quality, model behavior, decision policy, and operational monitoring, then asks how each layer changes under distribution shift, adversarial pressure, or incomplete information. It recommends evaluation by slices and repeated trials, explicit reject and escalation policies, preservation of data and reasoning lineage, and prospective monitoring tied to defined actions. The result is a research agenda for systems that are efficient enough to use but also bounded enough to audit. No new experiment is claimed; the article develops a comparative conceptual model and identifies tests that would make future empirical claims more credible.

1. Wang, Zijun, et al. "Trident: Dual-Stream APT Attribution over Heterogeneous Threat Knowledge Graphs." *Computers & Security* 171 (2026): 105094.
2. Li, Yuanhao, et al. "BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models." *arXiv preprint arXiv:2605.09134* (2026).
3. Liu, Shunqi, et al. "A New Playing Method of the Guessing Football Lottery." *IOP Conference Series: Materials Science and Engineering* 790.1 (2020): 012100.
4. Maher, M. J. "Modelling Association Football Scores." *Statistica Neerlandica*, vol. 36, no. 3, 1982, pp. 109-118.
5. Dixon, Mark J., and Stuart G. Coles. "Modelling Association Football Scores and Inefficiencies in the Football Betting Market." *Journal of the Royal Statistical Society: Series C*, vol. 46, no. 2, 1997, pp. 265-280.
6. Rue, Havard, and Oyvind Salvesen. "Prediction and Retrospective Analysis of Soccer Matches in a League." *Journal of the Royal Statistical Society: Series D*, vol. 49, no. 3, 2000, pp. 399-418.
7. Hvattum, Lars Magnus, and Halvard Arntzen. "Using ELO Ratings for Match Result Prediction in Association Football." *International Journal of Forecasting*, vol. 26, no. 3, 2010, pp. 460-470.
8. Brier, Glenn W. "Verification of Forecasts Expressed in Terms of Probability." *Monthly Weather Review*, vol. 78, no. 1, 1950, pp. 1-3.
9. Gneiting, Tilmann, and Adrian E. Raftery. "Strictly Proper Scoring Rules, Prediction, and Estimation." *Journal of the American Statistical Association*, vol. 102, no. 477, 2007, pp. 359-378.
10. Dawid, A. P. "The Well-Calibrated Bayesian." *Journal of the American Statistical Association*, vol. 77, no. 379, 1982, pp. 605-610.
11. Kelly, J. L., Jr. "A New Interpretation of Information Rate." *Bell System Technical Journal*, vol. 35, no. 4, 1956, pp. 917-926.
12. Constantinou, Anthony C., and Norman E. Fenton. "Solving the Problem of Inadequate Scoring Rules for Assessing Probabilistic Football Forecast Models." *Journal of Quantitative Analysis in Sports*, vol. 9, no. 3, 2013, pp. 209-222.
13. Ovadia, Yaniv, et al. "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift." *Advances in Neural Information Processing Systems*, vol. 32, 2019.