Journal of the American Medical Informatics Association, Vol 3, 208-215, Copyright © 1996 by American Medical Informatics Association

ARTICLES

Relationships among performance scores of four diagnostic decision support systems

ES Berner, JR Jackson and J Algina
Office of Educational Development, School of Medicine, University of Alabama at Birmingham 35294-2041, USA. eberner@uasom.meis.uab.edu

OBJECTIVE: To examine the relationships among different performance scores for each of four diagnostic decision support systems (DDSSs). Design: Intercorrelations among seven performance scores on a set of 105 cases for each of four DDSSs (DXplain, Iliad, Meditel, QMR) were computed. METHODS: The performance scores for each case reflected: 1) presence or absence of the case diagnosis in the DDSS knowledge base; 2) presence or absence of the correct diagnosis anywhere on the DDSS diagnosis list; 3) presence or absence of the correct diagnosis in the top ten diagnoses; 4) relevance of the DDSS diagnosis list; 5) comprehensiveness of the DDSS diagnosis list; 6) whether the DDSS suggested additional diagnoses to the experts' list; and 7) the length of the DDSS diagnosis list. Results: For all DDSSs, the two Correct Diagnosis scores (top ten and total list) were significantly related: 1) to the presence of the correct diagnosis in the knowledge base; 2) to the Comprehensiveness score; and 3) to each other. There were significant differences among the four DDSSs on the magnitude and/or direction of the relationships between: 1) the two Correct Diagnosis scores; 2) the Relevance and Length scores; and 3) the Relevance and Additional Diagnoses scores. CONCLUSION: The production of a correct diagnosis for a given case is not related to the number of diagnoses suggested by the DDSS and, across different DDSSs, is not consistently related to other measures of performance. These data indicate that multiple measures are needed to fully describe the performance of a DDSS.

This article has been cited by other articles:

				O. I. Ogunyemi, J. R. Clarke, N. Ash, and B. L. Webber Combining Geometric and Probabilistic Reasoning for Computer-based Penetrating-Trauma Assessment J. Am. Med. Inform. Assoc., May 1, 2002; 9(3): 273 - 282. [Abstract] [Full Text] [PDF]

				T. H. Payne Computer Decision Support Systems Chest, August 1, 2000; 118(2_suppl): 47S - 52S. [Abstract] [Full Text] [PDF]

				E. S. Berner, R. S. Maisiak, C. G. Cobbs, and O. D. Taunton Effects of a Decision Support System on Physicians' Diagnostic Performance J. Am. Med. Inform. Assoc., September 1, 1999; 6(5): 420 - 427. [Abstract] [Full Text] [PDF]

				E. S. Berner and R. S. Maisiak Influence of Case and Physician Characteristics on Perceptions of Decision Support Systems J. Am. Med. Inform. Assoc., September 1, 1999; 6(5): 428 - 434. [Abstract] [Full Text] [PDF]

				E. E. Westberg and R. A. Miller The Basis for Using the Internet to Support the Information Needs of Primary Care J. Am. Med. Inform. Assoc., January 1, 1999; 6(1): 6 - 25. [Abstract] [Full Text] [PDF]