148 - Construct Validity and Reliability of a Novel Assessment Tool after Nutrition Education

Poster Number: 148
Publication Number: 148.218

Jaclyn L. Albin, University of Texas Southwestern Medical School, Bedford, TX, United States; Bethany Agusala, University of Texas Southwestern Medical School, Dallas, TX, United States

Background: Medical education fails to prepare learners to address the profound role of diet in health outcomes. Furthermore, educators struggle to measure competency for nutrition counseling skills as they lack valid tools to appraise skill application in existing nutrition curricula.

Objective: We developed a rubric to assess student skills in nutrition counseling and evaluated its construct validity and reliability.

Design/Methods: After a hands-on Culinary Medicine nutrition elective, first-year medical students (MS) completed a series of four nutrition focused OSCEs (Objective Structured Clinical Examinations) which required MS to address common health concerns and provide relevant counseling. After each encounter, MS documented an assessment and plan (A&P). A psychometric analyst, a third-year medical student, and a clinician educator with nutrition expertise developed a novel rubric to evaluate the A&P in key domains of lifestyle counseling and motivational interviewing (Table 1). The analyst (R1) and two clinician educators (R2 and R3) applied the rubric to score MS progress notes as a marker of counseling provided. Statisticians then assessed degree of agreement via Inter-Rater Reliability (IRR) using the Kappa coefficient.

Results: 39 MS each completed OSCE cases and an associated A&P. Raters applied the rubric to each student’s four cases, averaging 76 seconds per case. For Exercise (Item 2, all cases) and Dietitian (Item 7, cases 3 and 4) on the rubric (Table 1), the weighted Kappa estimate for all three pairs indicated almost perfect agreement (0.8-1). For other items, clinicians R2 and R3 shared higher agreement at moderate (0.4-0.6) to substantial (0.6-0.8) for Diet (Item 1, cases 1, 2, 4), Objective (Item 5, cases 1, 3, 4), and Barriers (Item 6, all cases) and a range of agreement for Items 3-4.Conclusion(s): As educators advance competency-based learner assessment, innovators must design and promote validity of practical assessment tools. Our rubric is feasible to apply to MS documentation but shows a range of agreement. The analyst’s scoring (R1) differed from the clinicians (R2, R3) in most domains, highlighting possible importance of clinical experience in student assessment or reflecting inadequate calibration. Additional calibration and refinement of the tool could improve both reliability and ability to discriminate the breadth of MS performance. This practical rubric evaluates applied skills after nutrition education and shows promise as a feasible approach for nutrition education evaluation.

Table 1. Evaluation Rubric for Nutrition Counseling Encounter Documentation