1 - Assessing Treatment of Children with Attention-Deficit/Hyperactivity Disorder: A Novel Application of Natural Language Processing

Poster Number: 1
Publication Number: 1.414

Yair Bannett, Stanford University, Palo Alto, CA, United States; Jose Posada, Universidad del Norte, Barranquilla, Atlantico, Colombia; Rebecca M. Gardner, Stanford University, Palo Alto, CA, United States; Sonam Welekar, San Jose State University, Santa Clara, CA, United States; Lynne Huffman, Stanford University School of Medicine, Palo Alto, CA, United States; Heidi M. Feldman, Stanford University School of Medicine, Palo Alto, CA, United States

Background: Quality measures for child mental health disorders, including attention-deficit/hyperactivity disorder (ADHD), are critically needed. Current national ADHD quality measures rely on limited claims-based data that are disconnected from practice guidelines. Alternative methods require healthcare organizations to perform costly, labor-intensive chart reviews.

Objective: To develop and test a natural language processing classifier of electronic health records that captures pediatrician adherence to ADHD guidelines, which recommend parent training in behavior management as first-line for young children with ADHD.

Design/Methods: We extracted clinical notes of all office visits of children aged 4-6 years, seen >2 times in 2015-2019 in a community-based primary care network in California, who had >1 visit with an ICD-10 diagnosis of ADHD (cohort n=423). To create a “gold standard”, 2 pediatricians manually annotated notes of the first ADHD visit for each patient. Inter-annotator agreement was assessed for recommendation of behavioral treatment; disagreements (13%) were reconciled. We used a random subset of first-visit notes (n=296, 70%) to train and validate a classification algorithm based on Clinical Bidirectional Encoder Representations from Transformers (Clinical-BERT), a neural network-based technique for natural language processing of clinical text. The untrained test set of first-visit notes (n=127, 30%) was used to assess model performance compared to manual chart review. We then chose model thresholds to maximize sensitivity (recall) and minimize the false negative rate. We completed external validation by deploying the model on all other notes of ADHD or well-care visits in the same patient cohort (other notes=1,020); we annotated all notes the model classified as positive (n=50) and 5% of notes classified as negative (n=50).

Results: The model achieved acceptable performance (F1>0.75) classifying first ADHD visits (Table 1), capturing low rates of behavioral treatment recommendations (40% of visits). Following threshold selection, external validation yielded improved model performance with recall=0.92 (Table 2) and revealed that pediatricians recommended behavioral treatment in only 5% of non-first-ADHD visits.Conclusion(s): We demonstrated the utility of deploying a natural language processing algorithm on a large and variable set of clinical notes to assess pediatrician adherence to guidelines for ADHD treatment. This approach could enable scalable and continuous quality measurement of clinical care for ADHD and other chronic conditions leading to improved health care delivery and health outcomes.