A response to Marquis et al (2017) What is the error margin of your signature analysis?

Abstract

Marquis et al (2017) [What is the error margin of your signature analysis? Forensic Science International, 281, e1–e8] ostensibly presents a model of how to respond to a request from a court to state an “error margin” for a conclusion from a forensic analysis. We interpret the court’s request as an explicit request for meaningful empirical validation to be conducted and the results reported. Marquis et al (2017), however, recommends a method based entirely on subjective judgement and does not subject it to any empirical validation. We believe that much resistance to the adoption of the likelihood ratio framework is not to the idea of assessing the relative probabilities (or likelihoods) of the evidence under prosecution and defence hypotheses per se, but to what is perceived to be unwarranted subjective assignment of those probabilities. In order to maximize transparency, replicability, and resistance to cognitive bias, we recommend the use of methods based on relevant data, quantitative measurements, and statistical models. If the method is based on subjective judgement, the output should be empirically calibrated. Irrespective of the basis of the method, its implementation should be empirically validated under conditions reflecting those of the case at hand.