604 - Using Virtual Reality Simulations to Establish Validity Evidence for a Clinical Assessment Instrument of Behavioral Health Skills

Poster Number: 604
Publication Number: 604.118

Francis Real, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Dominick DeBlasio, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Nicholas Ollberding, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Kimberley Sikora, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; RACHEL HERBST, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Monica R. Whitehead, Cincinnati Children's Hospital Medical Center, Norwood, OH, United States; Brittany Rosen, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Andrea Meisman, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Lori E. Crosby, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Melissa Klein, Cincinnati Children's Hospital Medical Center ; University of Cincinnati, Cincinnati, OH, United States

Background: The development of entrustable professional activities has created calls to enhance behavioral health training. Assessing pediatric residents’ skills is challenging due to a lack of psychometrically valid instruments.

Objective: Our team developed and implemented an observation instrument to assess residents’ competencies related to behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI) though its reliable usage required expert raters (e.g., pediatric psychologists). Our purpose was to establish validity evidence for an adapted observation instrument practical for clinicians by using previously recorded virtual reality (VR) simulations.

Design/Methods: This study utilized data from a prior educational trial of a VR-based behavioral health curriculum for pediatric residents at a large, urban primary care center. Two group interviews were conducted with clinicians to modify the existing instrument. Next, recorded VR simulations (n=10) were scored to assess interrater reliability between a mid-level clinician with a junior clinicinian, senior clinician, and pediatric psychologist. Finally, this instrument was used by the mid-level clinician to evaluate performance of prior trial participants in a standardized VR simulation to assess its ability to discriminate between skill levels and correlate with findings demonstrated on the prior, more complex instrument, requiring psychologist raters.

Results: Modifications during group interviews included a change from frequency counts for items related to MI to the binary outcome of “present” or “absent". An item designating entrustment along an anchored 5-point scale was added (Figure 1). On interrater-reliability testing, the updated instrument demonstrated mostly substantial or near perfect agreement for items related to BHAG. MI items demonstrated a range of agreement (Table 1). Using 55 previously recorded standardized VR simulations of unique inividuals, the instrument discriminated between trained v untrained individuals in BHAG skills (p=0.002) and level of entrustment (p=0.001). Differences between groups in MI adherence did not demonstrate a significant difference on this instrument (p=0.095) as it did on the prior one requiring expert raters (p=0.04, abstract ID: 1171095).Conclusion(s): VR demonstrated potential as a novel approach for obtaining validity evidence to inform development of an instrument for direct observation of behavioral health skills.
Figure 1. Updated instrument for clinicians to assess residents' skills at delivering behavioral health anticipatory guidance
Table 1. Interrater reliability of items included on updated assessment instrument for clinicians