| Title: |
Patient Experiences in the Cochlear Implant Reddit Community: Comparing Human and Large Language Model Categorization. |
| Authors: |
Habib, Daniel R. S.; Depala, Kiran; Lin, Jack; Le, Samuel; McFall, Natalie; Dewan, Shiv S.; Huang, Justin; Habib, Michael W. S.; Bishay, Anthony E.; Siebor, Konrad; Babaoglu, Gizem; Chowdhury, Naweed I.; Moberly, Aaron C. |
| Source: |
American Journal of Audiology; Jun2026, Vol. 35 Issue 2, p487-496, 10p |
| Subject Terms: |
Social media; Cochlear implants; Predictive tests; Generative artificial intelligence; Community support; Qualitative research; Cronbach's alpha; Health; Information resources; Natural language processing; Descriptive statistics; Thematic analysis; Medical coding; Statistics; Comparative studies; Social support; Data analysis software; Automation; Patients' attitudes; Sensitivity & specificity (Statistics); Inter-observer reliability |
| Abstract: |
Purpose: Although some work has leveraged automated analyses of online communities to gain cochlear implant (CI) patient insights, there remains a gap in comparing human versus automated analysis of the nuanced, real-world experiences patients share outside clinical settings. This study characterizes experiences within the r/Cochlearimplants Reddit community and compares human to large language model (LLM) performance in annotating posts. Method: Using reflexive thematic analysis, 996 publicly available r/Cochlearimplants posts (October 2024--June 2025) were manually coded and consolidated into themes. Three LLMs--OpenAI o3, Gemini 2.5 Pro, and Claude Sonnet 4--were prompted with the posts and human-generated codebook to perform post categorization. Model performance was evaluated against human coding using Cohen's kappa, percent agreement, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and time. Results: Five themes emerged. Community engagement and support (N = 944, 94.8%) frequently involved eliciting advice (N = 721, 72.4%), seeking shared experiences (N = 249, 25.0%), and sharing negative experiences (N = 247, 24.8%). Other themes included the medical/surgical journey (N = 463, 46.5%), device/ technical issues (N = 343, 34.4%), daily life/adjustments (N = 236, 23.7%), and media/outreach (7.2%, N = 72). OpenAI o3 and Gemini 2.5 Pro achieved the highest interrater reliability with human annotators (Κ = .35 and Κ = .34, respectively). OpenAI o3 had higher sensitivity (46.7%) but lower specificity (90.4%) than Gemini 2.5 Pro, which had the highest specificity (93.4%) but lower sensitivity (38.0%). Claude Sonnet 4 showed the lowest agreement (Κ = .25) and PPV (30.9%). Compared to human annotation requiring 52 hr across all annotators, each LLM required less than 20 min. Conclusions: Reddit posts revealed rich discourse across CI topics. LLMs demonstrated fair agreement with human coders and can quickly aid in large-scale qualitative analysis. Although careful model selection and human expertise remain essential for accurate interpretation, LLM annotation shows potential for real-time monitoring of patient concerns to inform counseling, rehabilitation strategies, and iterative device design. [ABSTRACT FROM AUTHOR] |
| : |
Copyright of American Journal of Audiology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Database: |
Complementary Index |