In the context of forensic casework, are there meaningful metrics of the degree of calibration?

Abstract

Forensic-evaluation systems should output likelihood-ratio values that are well calibrated. If they do not, their output will be misleading. Unless a forensic-evaluation system is intrinsically well-calibrated, it should be calibrated using a parsimonious parametric model that is trained using calibration data. The system should then be tested using validation data. Metrics of degree of calibration that are based on the pool-adjacent-violators (PAV) algorithm recalibrate the likelihood-ratio values calculated from the validation data. The PAV algorithm overfits on the validation data because it is both trained and tested on the validation data, and because it is a non-parametric model with weak constraints. For already-calibrated systems, PAV-based ostensive metrics of degree of calibration do not actually measure degree of calibration; they measure sampling variability between the calibration data and the validation data, and overfitting on the validation data. Monte Carlo simulations are used to demonstrate that this is the case. We therefore argue that, in the context of casework, PAV-based metrics are not meaningful metrics of degree of calibration; however, we also argue that, in the context of casework, a metric of degree of calibration is not required.

Publication DOI: https://doi.org/10.1016/j.fsisyn.2021.100157
Divisions: College of Engineering & Physical Sciences > Computer Science
College of Engineering & Physical Sciences
School of Social Sciences & Humanities > Aston Institute for Forensic Linguistics
Additional Information: © 2021 The Author. This is an open access article under the CC BY-NC-ND license Funding: This research was supported by Research England's Expanding Excellence in England Fund as part of funding for the Aston Institute for Forensic Linguistics 2019–2022.
Uncontrolled Keywords: Calibration,Forensic inference and statistics,Likelihood ratio,Metric,Pathology and Forensic Medicine,Law
Full Text Link:
Related URLs: https://www.sci ... 0279?via%3Dihub (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2021-06-12
Accepted Date: 2021-06-08
Authors: Morrison, Geoffrey Stewart (ORCID Profile 0000-0001-8608-8207)

Export / Share Citation


Statistics

Additional statistics for this record