278 - Validity Evidence for an Antenatal Counseling Skills Measurement Tool

Poster Number: 278
Publication Number: 278.331

Michelle J. Bartlett, University of Washington School of Medicine, Seattle, WA, United States; Rachel Umoren, University of Washington School of Medicine, Seattle, WA, United States; Jennifer C. Kett, Seattle Children's Hospital, Seattle, WA, United States; Josephine H. Amory, University of Washington, Seattle, WA, United States; Trang Huynh, Oregon Health and Science University, Portland, OR, United States; Amanda J. Kim, Oregon Health & Science University, Portland, OR, United States; Amanda K. Stiffler, University of Washington School of Medicine, Mill Valley, CA, United States; Rossella Mastroianni, University of Washington School of Medicine, Division of Neonatology, bellevue, WA, United States; Ellie Ficco, University of Washington School of Medicine, Seattle, WA, United States; Heather French, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, United States; Megan Gray, University of Washington School of Medicine, Seattle, WA, United States

Background: Antenatal counseling for parents in the setting of expected preterm delivery is an important component of pediatric training. However, healthcare professionals receive a variable amount and quality of formal training in these communication skills and there are no validated tools to measure the quality of antenatal counseling encounters.

Objective: This study evaluated and discussed validity of a practical tool to assess antenatal counseling skills, provide feedback, and offer coaching: the Antenatal Counseling Milestones Scale (ACoMS).

Design/Methods: Experts in antenatal counseling developed an anchored milestone-based tool to evaluate observable skills (Fig 1). Study participants with a range of antenatal counseling skills (residents, fellows, and attendings) were recruited to participate in simulated counseling sessions both in person and via video with standardized patient actors presenting with preterm labor at 23 weeks gestation. Two faculty observers scored each session independently using the ACoMS. Participants completed an ACoMS self-assessment, demographic and feedback survey. Validity was measured with weighted kappas for inter-rater agreement and Kruskal-Wallis tests for milestone levels between degrees of expertise in counseling. Cronbach’s alpha was measured for item consistency.

Results: 42 participants completed observed counseling sessions (Table 1). Of the 17 items included in the tool, 13 items were statistically significant with scores scaling with level of training. The majority of elements had fair-moderate agreement between raters (Table 2). Cronbach’s alpha demonstrated that there was relatively high internal consistency amongst items with all greater than 0.90.Conclusion(s): The ACoMS rubric internal structure demonstrates greater than fair inter-rater reliability and high reliability amongst items. Content validity is supported by the scale’s ability to discern level of training. Rater training and adjustments to the tool to drop items that do not accurately discern skill level may further improve reliability. Application of the ACoMS to clinical encounters is needed to determine utility in clinical practice.
Michelle Bartlett CVBartlett_CV-New.pdf
Table 2. Validity evidence as demonstrated by item scaling amongst skill level (kruskal wallis p-value), interrater reliability (weight kappa), and internal consistency (Cronbach’s alpha).* Included in this element was preparing the room and sitting with the parents which was not able to be scored during video conference simulations.
^ Element added after the first 11 participants based on feedback from faculty observers and participants that these were key behaviors not captured in the original tool.