589 - Validity Evidence for a Central Line Insertion Competency Checklist for Pediatric Critical Care Fellows and Its Association with Entrustment and Global Rating Scales

Poster Number: 589
Publication Number: 589.117

Samuel A. Rosenblatt, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, United States; Ben D. Albert, Harvard Medical School, Boston, MA, United States; Alvaro Coronado Munoz, University of Texas Health science center at Houston, Houston, TX, United States; Kim Derespina, Mount Sinai Kravis Children’s Hospital, New York, NY, United States; Ryan J. Good, University of Colorado School of Medicine, Aurora, CO, United States; Jennifer King, Monroe Carell Jr. Children's Hospital at Vanderbilt, Nashville, TN, United States; Amanda B. Levin, Johns Hopkins University School of Medicine, Bethesda, MD, United States; Katherine Mason, The Warren Alpert Medical School of Brown University, Providence, RI, United States; Richard Mink, The Lundquist Institute for Biomedical Innovation, Torrance, CA, United States; Jennifer C. Munoz Pareja, Pediatrics, Miami, FL, United States; Katie R. Nielsen, University of Washington School of Medicine, Seattle, WA, United States; Tara Petersen, Medical College Of Wisconsin/Children’s Wisconsin, Milwaukee, WI, United States; Rachel S. Poeppelman, University of Minnesota Masonic Children's Hospital, Edina, MN, United States; Jennifer Schuette, Johns Hopkins Children's Center, Baltimore, MD, United States; Sushant Srinivasan, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States; Hossein Tcharmtchi, Baylor College of Medicine, Houston, TX, United States; David A. Turner, American Board of Pediatrics, Chapel Hill, NC, United States; Jason Werner, Saint Louis University, St. Louis, MO, United States; Allison Whalen, Medical University of South Carolina, Charleston, SC, United States; Adrian D. Zurca, Pennsylvania State University College of Medicine, Hershey, PA, United States; Alan Schwartz, University of Illinois College of Medicine, Chicago, IL, United States; Donald Boyer, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, United States

Background: Central Venous Catheter (CVC) placement is an essential skill that Pediatric Critical Care Medicine (PCCM) fellows must achieve during training. There are multiple procedural competency assessment tools for CVC insertion and no standardized approach for use. A direct observation checklist tool (CT) to assess competency in CVC placement in children was previously developed and validated in simulated encounters but not in clinical environments.

Objective: This study aimed to understand how the observation CT performs in a clinical environment and compare it with two other competency assessment tools currently in use, a global rating scale (GRS) and an entrustment scale (ES).

Design/Methods: Prospective, multi-site, observational study with a 6-month study period (7/21-1/22). With each CVC inserted, supervisors completed 3 assessments of competency (CT, GRS, ES). The CT included 15 discrete steps with dichotomous (Complete/Not Complete) options; GRS was a dichotomous (Competent/Not Competent) assessment; ES was a 5-level supervision rating ranging from ‘observe only’ to ‘able to perform procedure unsupervised’. CT inter-item reliability was evaluated with Cronbach’s alpha and CT, GRS, and ES assessments were compared using partial correlations adjusting for clustering of observations in fellows and fellows in programs. Mixed-effects linear, ordinal, and logistic regression were used to model the effect of fellow year on scales as appropriate.

Results: 17 sites participated with 159 CVC insertions completed by 87 unique fellows (Table 1). Overall, the CT had an acceptable internal reliability (alpha=0.65 [95% CI 0.56-0.72]). The CT had minimal to no correlation with the other assessments (GRS R=0.34, ES R=0.19), but the GRS and ES (R=0.59) were moderately correlated (Figure 1). There was no correlation between any specific CT item and competency as assessed by GRS. Fellow training year was associated with competency by CT (Table 2) from year 1 to 2 (p=0.007) but not year 2 to 3 (p=0.516). For GRS and ES, training year was associated with competency in year 1 to 2 (p < 0.001) and year 2 to 3 (p < 0.001).Conclusion(s): The checklist tool performed acceptably during real-world competency assessment and correlated with fellow year of training only from year 1 to 2 and not year 2 to 3, perhaps reflecting its strength in assessing competency for novice trainees. Current assessment tools (ES and GRS) did not correlate well with the CT. Further evaluation of these assessment tools and their correlation is warranted to determine optimal approach to competency assessment, which may vary by level of trainee experience.
Table 1Demographics of Fellow and Faculty Participants
Figure 1Heat Map of Correlation Between Three Supervisor Rating Scales - Checklist Tool, Entrustment Scale, and Global Rating Scale