| Title: |
Large language models accurately identify immunosuppression in intensive care unit patients |
| Authors: |
Guggilla, Vijeeth; Kang, Mengjia; Bak, Melissa J; Tran, Steven D; Pawlowski, Anna; Nannapaneni, Prasanth; Rasmussen, Luke V; Schneider, Daniel; Donnelly, Helen K; Agrawal, Ankit; Liebovitz, David; Misharin, Alexander V; Budinger, G R Scott; Wunderink, Richard G; Walunas, Theresa L; Gao, Catherine A; Hauser, Alan R; Peltekian, Alec; Wolfe, Alexis Rose; Szabo, Alison L; Choudhary, Alok; Ludwig, Amy; Moghadam, Anahid Amani; Yeldandi, Anjana V; Bharat, Ankit; Pawlowski, Anna E; Joudi, Anthony M; Tambe, Arjun Prakash; Smith-Nunez, Ashley J; Singer, Benjamin D; Ulrich, Benjamin J; Tran, Betty; Gottardi, Cara J; Qi, Chao; Pickens, Chiagozie O; Schroedl, Clara J; Meza, Daniel; Garcia, Dulce Sarai; Ozer, Egon A; Gusman, Elen; Shanes, Elisheva D; Leibenguth, Emily M; Olson, Emily M; Hartmann, Erica Marie; Korth, Erin A; Diaz, Estefani |
| Contributors: |
National Institutes of Health; Northwestern University Clinical and Translational Sciences Institute; Simpson Querrey Lung Institute for Translational Science; Canning Thoracic Institute of Northwestern Medicine; Chicago Biomedical Consortium; Northwestern University Dixon Translational Science; Veterans Administration; Gilead Sciences; American Thoracic Society Unrestricted |
| Source: |
Journal of the American Medical Informatics Association ; volume 32, issue 12, page 1888-1898 ; ISSN 1067-5027 1527-974X |
| Publisher Information: |
Oxford University Press (OUP) |
| Publication Year: |
2025 |
| Description: |
Objective Rule-based structured data algorithms and natural language processing (NLP) approaches applied to unstructured clinical notes have limited accuracy and poor generalizability for identifying immunosuppression. Large language models (LLMs) may effectively identify patients with heterogenous types of immunosuppression from unstructured clinical notes. We compared the performance of LLMs applied to unstructured notes for identifying patients with immunosuppressive conditions or immunosuppressive medication use against 2 baselines: (1) structured data algorithms using diagnosis codes and medication orders and (2) NLP approaches applied to unstructured notes. Materials and Methods We used hospital admission notes from a primary cohort of 827 intensive care unit (ICU) patients at Northwestern Memorial Hospital and a validation cohort of 200 ICU patients at Beth Israel Deaconess Medical Center, along with diagnosis codes and medication orders from the primary cohort. We evaluated the performance of structured data algorithms, NLP approaches, and LLMs in identifying 7 immunosuppressive conditions and 6 immunosuppressive medications. Results In the primary cohort, structured data algorithms achieved peak F1 scores ranging from 0.30 to 0.97 for identifying immunosuppressive conditions and medications. NLP approaches achieved peak F1 scores ranging from 0 to 1. GPT-4o outperformed or matched structured data algorithms and NLP approaches across all conditions and medications, with F1 scores ranging from 0.51 to 1. GPT-4o also performed impressively in our validation cohort (F1 = 1 for 8/13 variables). Discussion LLMs, particularly GPT-4o, outperformed structured data algorithms and NLP approaches in identifying immunosuppressive conditions and medications with robust external validation. Conclusion LLMs can be applied for improved cohort identification for research purposes. |
| Document Type: |
article in journal/newspaper |
| Language: |
English |
| DOI: |
10.1093/jamia/ocaf141 |
| Availability: |
https://doi.org/10.1093/jamia/ocaf141; https://academic.oup.com/jamia/article-pdf/32/12/1888/64341335/ocaf141.pdf |
| Rights: |
https://creativecommons.org/licenses/by-nc-nd/4.0/ |
| Accession Number: |
edsbas.D642757D |
| Database: |
BASE |