172 - Testing inter-rater reliability of a novel checklist for evaluating the shared decision making process in consultation for extremely preterm delivery

Poster Number: 172
Publication Number: 172.233

Michael G. Guindon, Uniformed Services University of the Health Sciences F. Edward Hebert School of Medicine, San Antonio, TX, United States; Dalia M. Feltman, NorthShore University HealthSystem Evanston Hospital, Burr Ridge, IL, United States; Jeanne Krick, Brooke Army Medical Center, San Antonio, TX, United States; Elizabeth Okonek, Brooke Army Medical Center, Fort Sam houston, TX, United States; Kaitlyn T. Mullin, Brooke Army Medical Center, San Antonio, TX, United States; Carrie Litke, Brooke Army Medical Center, San Antonio, VA, United States; Uchenna E. Anani, Vanderbilt University Medical Center, Nashville, TN, United States; Peter D. Murray, University of Virginia School of Medicine, Charlottesville, VA, United States; Christopher Mattson, Comer Children's Hospital at University of Chicago Medical Center, Chicago, IL, United States

Background: Parents facing extremely preterm delivery must partner with neonatologists and obstetricians to make decisions for the care of their child. Shared decision making (SDM) between parents and the medical team is recommended to facilitate integration of parental values with physician’s medical knowledge to develop the best course of action, but is often difficult to execute. An objective scoring method may be useful to aid in assessment of trainee performance and parent satisfaction with counseling.

Objective: To develop and evaluate a novel checklist to assess SDM practices of neonatologists during antenatal counseling for extremely preterm delivery.

Design/Methods: The novel checklist was developed after review of the literature and based on the researchers' clinical experience. Its two components are information topics and SDM process. The SDM component of the checklist has 3 subparts of observed behaviors (Table 1) rated as “yes/no.” Fourteen neonatologists from 4 U.S. centers were video-recorded performing antenatal consultations for a mock patient facing delivery at 22 weeks’ gestation. Each consultation was evaluated using the novel checklist by 2 “experts” (researchers involved in checklist development) and 3 “novices” (reviewers receiving no specialized checklist training). Intraclass correlation coefficient (ICC) and Cronbach’s alpha were calculated to assess inter-rater agreement and internal consistency, respectively. 

Results: Checklist component ICCs are shown for expert and novice raters in Table 2. Inter-rater agreement was higher for information topics. SDM process had substantial inter-rater agreement amongst experts (0.75), but poor agreement amongst novices (0.37). Two of three subcategories included in the checklist demonstrated moderate to good reliability: Setting up the consult (ICC 0.66) and Conveying information (0.79). ICC was lowest for Decision making in both groups, though expert (0.374) was higher than novice (0.178). Internal consistency of the checklist was high (Cronbach’s alpha = 0.87). Novice raters identified unclear checklist items and endorsed providing structured training on checklist use. Conclusion(s): Our novel checklist for evaluating SDM in antenatal consultation demonstrated strong inter-rater agreement amongst those with training in its content and high internal consistency. Low-ICC items will be revised before rating future mock consultations. A more structured training for novice checklist users will enable its wide use for building antenatal counseling skills. Comparing results of this checklist with parental responses to test external validity is planned.
Table 1. Shared Decision Making Process Evaluation ChecklistShared decision making checklist demonstrating components of prenatal counseling
Table 2: Inter-rater reliability between two raters (2 expert or 2 novice) for 14 mock consultationsICC closest to 1 denotes greatest inter-rater reliability. Reliability was highest and most similar between different rater types for the information topics component of the checklist and lowest for the decision making subpart of SDM process component, but remained higher in expert ratings.