| Title: |
Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models’ feasibility in clinical decision-making |
| Authors: |
Sblendorio E.; Dentamaro V.; Lo Cascio A.; Germini F.; Piredda M.; Cicolini G. |
| Contributors: |
Sblendorio, E.; Dentamaro, V.; Lo Cascio, A.; Germini, F.; Piredda, M.; Cicolini, G. |
| Publication Year: |
2024 |
| Collection: |
Università degli Studi di Bari Aldo Moro: CINECA IRIS |
| Subject Terms: |
AI routine integration; Clinical decision making; Healthcare innovation; LLM's feasibility; Methodology; Multi-parametric analysi; Multidisciplinary approach; Nursing informatic; Safety |
| Description: |
Background: Recent enhancements in Large Language Models (LLMs) such as ChatGPT have exponentially increased user adoption. These models are accessible on mobile devices and support multimodal interactions, including conversations, code generation, and patient image uploads, broadening their utility in providing healthcare professionals with real-time support for clinical decision-making. Nevertheless, many authors have highlighted serious risks that may arise from the adoption of LLMs, principally related to safety and alignment with ethical guidelines. Objective: To address these challenges, we introduce a novel methodological approach designed to assess the specific feasibility of adopting LLMs within a healthcare area, with a focus on clinical nursing, evaluating their performance and thereby directing their choice. Emphasizing LLMs’ adherence to scientific advancements, this approach prioritizes safety and care personalization, according to the “Organization for Economic Co-operation and Development” frameworks for responsible AI. Moreover, its dynamic nature is designed to adapt to future evolutions of LLMs. Method: Through integrating advanced multidisciplinary knowledge, including Nursing Informatics, and aided by a prospective literature review, seven key domains and specific evaluation items were identified as follows: 1. State of the Art Alignment & Safety. 2. Focus, Accuracy & Management of Prompt Ambiguity. 3. Data Integrity, Data Security, Ethics & Sustainability, in accordance with OECD Recommendations for Responsible AI. 4. Temporal Variability of Responses (Consistency) 5. Adaptation to specific standardized terminology and Classifications for healthcare professionals. 6. General Capabilities: Post User Feedback Self-Evolution Capability and Organization in Chapters. 7. Ability to Drive Evolution in Healthcare.A Peer Review by experts in Nursing and AI was performed, ensuring scientific rigor and breadth of insights for an essential, reproducible, and coherent methodological ... |
| Document Type: |
article in journal/newspaper |
| Language: |
English |
| Relation: |
info:eu-repo/semantics/altIdentifier/pmid/38810498; info:eu-repo/semantics/altIdentifier/wos/WOS:001247894300001; volume:188; journal:INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS; https://hdl.handle.net/11586/487493 |
| DOI: |
10.1016/j.ijmedinf.2024.105501 |
| Availability: |
https://hdl.handle.net/11586/487493; https://doi.org/10.1016/j.ijmedinf.2024.105501 |
| Accession Number: |
edsbas.F7D6E603 |
| Database: |
BASE |