578 - An exclusively narrative format for fellow assessment of Pediatric Subspecialty Milestones

Poster Number: 578
Publication Number: 578.117

Stacy Cooper, Johns Hopkins University, Baltimore, MD, United States; Richard Mink, The Lundquist Institute for Biomedical Innovation, Torrance, CA, United States

Background: Effective assessment and feedback are critical components of competency-based education, but many programs struggle in these areas. To improve trainee feedback and formative assessment, our pediatric hematology-oncology fellowship instituted a novel approach that bundles assessment and feedback into an email-based narrative format at the end of each 1-2 weeks on clinical service.

Objective: To evaluate the efficacy of our narrative assessment program on trainee feedback.

Design/Methods: Previously, face-to-face feedback occurred after time on service, with the attending completing an online assessment based on the milestones with comments optional. In 2016, a new approach was instituted. After face-to-face fellow-led feedback occurs, the faculty member summarizes the session using a written template (Fig 1), and sends it by email to the trainee and incoming attending. To evaluate efficacy, a survey was sent to fellows who graduated in 2015 and 2016, fellows who started fellowship in 2018 and 2019 and faculty asking about satisfaction with assessment and feedback.

Results: Survey response rates were 69% (9/13) for alumni, 91% (11/12) for current fellows, and 73% (22/30) for faculty. Current fellows and faculty rated their satisfaction with feedback in the new system (9.0/10 and 7.7/10, respectively) much higher than that compared with alumni (4.7/10). The use of narratives improved trainee ability to identify strengths and weaknesses (Fig 2) and promoted face-to-face feedback (Fig 3). Alumni noted the prior approach was “minimally helpful”, “largely meaningless”, and “brief and unsubstantial” whereas current trainees described the narrative system as “a helpful way to consistently get feedback and build on prior evaluations” and a tool to “hold both the trainee and attending accountable for trainee improvement”. 95% (21/22) of faculty preferred the newer system stating that it was “a marked improvement” and that it “takes longer but that’s only because feedback only rarely actually happened before.” Program compliance with “satisfaction with feedback” on the ACGME trainee survey improved from a mean of 76% (2014-2016) to 92% (2019-2021). Over the most recent academic year, 78% of faculty sent the email narrative within 3 days of finishing time on service. Conclusion(s): TThese data suggest that exclusively narrative faculty assessments are feasible with high compliance and is preferred by trainees and faculty. It provides a vehicle for accountability and normalizes constructive criticism. Further work is needed to determine if our experience can be replicated in other fellowships.
Figure 1: Narrative email templateEmail template completed by faculty after ending time on service with a fellow.
Figure 2: Trainee perception of ability to identify strengths and weaknessesSurvey results comparing alumni and current fellows' ability to identify strengths and weaknesses.