2022
Maddisen Neuman
Atlanta, GA
Authors: Maddisen Neuman, Callan Hundl, Aimee Grimaldi, Donna Eudaley, Darrell Stein, Peter Stout
Learning Objective: This presentation will inform attendees on preliminary findings from a blind testing program in the Firearms section at the Houston Forensic Science Center. Trends in the data, as well as benefits, limitation, and future directions of the blind testing program will be discussed.
Open proficiency tests created by external vendors are used to meet accreditation requirements and demonstrate examiner competence; however, the tests are limited in the ability to monitor the entire quality management system from evidence submission to reporting of results. As such, blind proficiency testing has been considered a more precise testing an individual’s accuracy (1).
In December 2015, the Houston Forensic Science Center (HFSC) implemented a blind quality control (blind QC) program in firearms examination. The intent of the blind QC program is to supplement open proficiency tests by creating mock cases that are submitted to the Firearms section as routine casework, with the expectation that the examiners are unaware that they are receiving test cases. Blind QC cases are created by Firearms section management and submitted by members of the Quality Division. Firearms section management evaluates the mock evidence prior to submission to determine the ground truth and expected results. Once the case is complete, the Firearms section management reviews the results to determine satisfactory completion. A satisfactory result may include: 1) a result that conforms to the known ground truth, or 2) a result that does not necessarily conform to the known ground truth but is technically sound (i.e., a known elimination/identification that is reported as inconclusive based on the applicable standards in the field). A firearms examiner can choose to conclude inconclusive if the item does not contain the quality or quantity of information needed to include or exclude from another item. Fragments, bullet cores, and poor-quality items are expected to be reported as unsuitable or insufficient regardless of ground truth based on availability of microscopic characteristics.
Fifty-one blind QC cases were reported between December 2015 and June 2021 resulting in 570 sufficiency and comparison conclusions. No eliminations were reported for true matching pairs, and no identifications were reported for true non-matching pairs; thus, all results were considered satisfactory. In 40.3% (n=225) of the comparisons, the ground truth result was either elimination or identification, but an inconclusive decision was reached. Variables such as the examiners assigned to the case, the training program, the examiner experience level, and the intended complexity of the case did not significantly contribute to the inconclusive results. The main factor contributing to the inconclusive conclusions was the type of evidence compared. Bullets were the main contributors to inconclusive results at a rate of 61.8% (n=168) compared to cartridge cases at a rate of 21.5% (n=57).
While a blind testing program can be challenging to implement, HFSC’s blind QC program demonstrates that the quality management system and procedures used by the Firearms section can obtain accurate and reliable results and provides examiners added confidence in court. Additionally, the blind QC program can be tailored to target specific research questions and provide opportunities for collaboration with other laboratories and researchers.
References: 1. Committee on Identifying the Needs of the Forensic Sciences Community, National Research Council. Strengthening forensic science in the United States: A path forward. Washington, DC, 2009.