This Article Full Text Full Text (PDF) Data Supplement All Versions of this Article: M2493v1 15/2/227    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Verduijn, M. Articles by de Mol, B. A.J.M. Search for Related Content PubMed PubMed Citation Articles by Verduijn, M. Articles by de Mol, B. A.J.M.

Individual and Joint Expert Judgments as Reference Standards in Artifact Detection

^a Department of Medical Informatics, Academic Medical Center, Amsterdam, The Netherlands
^b Department of Intensive Care Medicine, Academic Medical Center, Amsterdam, The Netherlands
^c Department of Cardio-thoracic Surgery, Academic Medical Center, Amsterdam, The Netherlands
^d Department of Biomedical Engineering, University of Technology, Eindhoven, The Netherlands.

^_* Correspondence: Marion Verduijn, PhD, Department of Medical Informatics, Academic Medical Center (AMC), P.O. Box 22700, 1100 DE Amsterdam, the Netherlands (Email: m.verduijn{at}amc.uva.nl).

Received for publication: 04/30/07; accepted for publication: 12/10/07.

Objective: To investigate the agreement among clinical experts in their judgments of monitoring data with respect to artifacts, and to examine the effect of reference standards that consist of individual and joint expert judgments on the performance of artifact filters.

Design: Individual judgments of four physicians, a majority vote judgment, and a consensus judgment were obtained for 30 time series of three monitoring variables: mean arterial blood pressure (ABPm), central venous pressure (CVP), and heart rate (HR). The individual and joint judgments were used to tune three existing automated filtering methods and to evaluate the performance of the resulting filters.

Measurements: The interrater agreement was calculated in terms of positive specific agreement (PSA). The performance of the artifact filters was quantified in terms of sensitivity and positive predictive value (PPV).

Results: PSA values between 0.33 and 0.85 were observed among clinical experts in their selection of artifacts, with relatively high values for CVP data. Artifact filters developed using judgments of individual experts were found to moderately generalize to new time series and other experts; sensitivity values ranged from 0.40 to 0.60 for ABPm and HR filters (PPV: 0.57–0.84), and from 0.63 to 0.80 for CVP filters (PPV: 0.71–0.86). A higher performance value for the filters was found for the three variable types when joint judgments were used for tuning the filtering methods.

Conclusion: Given the disagreement among experts in their individual judgment of monitoring data with respect to artifacts, the use of joint reference standards obtained from multiple experts is recommended for development of automatic artifact filters.