Gonarthrosis Advisor vs ChatGPT-5: quality and readability of artificial intelligence-generated patient education for knee osteoarthritis.
| Title: | Gonarthrosis Advisor vs ChatGPT-5: quality and readability of artificial intelligence-generated patient education for knee osteoarthritis. |
|---|---|
| Authors: | Öner SK; Department of Orthopaedics and Traumatology, Kütahya Health Sciences University Kütahya City Hospital, Kütahya, Türkiye.; Demirkiran ND; Department of Orthopaedics and Traumatology, Kütahya Health Sciences University Kütahya City Hospital, Kütahya, Türkiye.; Canlı EA; Department of Orthopaedics and Traumatology, Kütahya Health Sciences University Kütahya City Hospital, Kütahya, Türkiye.; Bilir A; Department of Orthopaedics and Traumatology, Kütahya Health Sciences University Kütahya City Hospital, Kütahya, Türkiye. |
| Source: | Acta orthopaedica et traumatologica turcica [Acta Orthop Traumatol Turc] 2026 Apr 08; Vol. 60 (2). Date of Electronic Publication: 2026 Apr 08. |
| Publication Type: | Journal Article; Comparative Study |
| Language: | English |
| Journal Info: | Publisher: Turkish Association of Orthopaedics and Traumatology with publishing services by AVES Country of Publication: Turkey NLM ID: 9424806 Publication Model: Electronic Cited Medium: Internet ISSN: 2589-1294 (Electronic) Linking ISSN: 1017995X NLM ISO Abbreviation: Acta Orthop Traumatol Turc Subsets: MEDLINE |
| Imprint Name(s): | Publication: -: [Ankara, Turkey] : Turkish Association of Orthopaedics and Traumatology with publishing services by AVES; Original Publication: İstanbul : Türk Ortopedi ve Travmatoloji Derneği |
| MeSH Terms: | Osteoarthritis, Knee*/therapy ; Patient Education as Topic*/methods ; Patient Education as Topic*/standards ; Comprehension* ; Artificial Intelligence*; Humans ; Large Language Models ; Reproducibility of Results ; Turkey ; Generative Artificial Intelligence |
| Abstract: | Objective: The primary objective of this study is to compare the quality and readability of patient education materials generated by a general-purpose large language model (ChatGPT-5) versus a guideline-based, fine-tuned model (the Gonarthrosis Advisor). The study aims to quantify the performance gains achieved through domain-specific, fine-tuning, and reinforcement learning using osteoarthritis clinical guidelines. Methods: Thirty frequently asked patient questions regarding knee osteoarthritis were compiled from Google's "People Also Ask" feature and outpatient clinical observations in May 2025. Responses were generated in Turkish by both the Gonarthrosis Advisor and ChatGPT-5 to reflect real-world patient education materials. Content quality was assessed by 2 independent orthopedic surgeons who were blinded to model identity to minimize bias. Both reviewers were co-authors of the article yet did not participate in model construction or data analysis. The assessments utilized the DISCERN instrument, a validated 16-item measure for evaluating the reliability and quality of treatment-related information. Readability was analyzed using the Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), and Turkish specific indices (Ateşman and Çakır-Demir). For comparability, English-based indices were applied to translated versions of the responses, whereas Turkish indices were applied to the original texts. All responses were anonymized and randomized prior to evaluation. Model identifiers were removed, and each response was presented in a standardized format to ensure blinding of reviewers. Inter-rater reliability was measured using Cronbach's α. Normality assumptions were tested, and Wilcoxon signed-rank tests were used for statistical comparisons. As no human subjects or personal data were involved, ethical approval was not required. Results: Mean DISCERN scores corresponded to the "good" category (66.4) for the Gonarthrosis Advisor and the 'moderate' category (54.2) for ChatGPT-5, according to established cut-off thresholds. The Gonarthrosis Advisor achieved significantly higher DISCERN scores than ChatGPT-5 (66.4 ± 4.8 vs. 54.2 ± 5.6; P < .001) with high inter-rater reliability (Cronbach's α = 0.86). Readability metrics favored the Gonarthrosis Advisor across all indices: lower FKGL (7.8 ± 0.7 vs. 9.6 ± 0.9) and higher FRES (54.3 ± 3.4 vs. 46.7 ± 3.7), Ateşman (92.0 ± 4.2 vs. 84.3 ± 4.9), and Çakır-Demir (111.7 ± 5.1 vs. 106.9 ± 5.4) scores (all P |
| Entry Date(s): | Date Created: 20260627 Date Completed: 20260627 Latest Revision: 20260627 |
| Update Code: | 20260628 |
| DOI: | 10.5152/j.aott.2026.25618 |
| PMID: | 42364231 |
| Database: | MEDLINE |
Journal Article; Comparative Study