| Title: |
A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks |
| Authors: |
Labrak, Yanis; Rouvier, Mickaël; Dufour, Richard |
| Contributors: |
Laboratoire Informatique d'Avignon (LIA); Avignon Université (AU)-Centre d'Enseignement et de Recherche en Informatique - CERI; Zenidoc; Traitement Automatique du Langage Naturel (LS2N - équipe TALN); Laboratoire des Sciences du Numérique de Nantes (LS2N); Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique); Institut Mines-Télécom Paris (IMT)-Institut Mines-Télécom Paris (IMT)-NANTES UNIVERSITÉ - École Centrale de Nantes (Nantes Univ - ECN); Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes université - UFR des Sciences et des Techniques (Nantes univ - UFR ST); Nantes Université - pôle Sciences et technologie; Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes Université - pôle Sciences et technologie; Nantes Université (Nantes Univ)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique); Nantes Université (Nantes Univ); Nicoletta Calzolari; Min-Yen Kan; ANR-23-IAS1-0005,MALADES,Grands modèles de langue adaptables et souverains pour le domaine médical français(2023) |
| Source: |
Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024) ; https://hal.science/hal-04470883 ; Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024), Nicoletta Calzolari; Min-Yen Kan, May 2024, Torino, Italy |
| Publisher Information: |
CCSD |
| Publication Year: |
2024 |
| Collection: |
Université de Nantes: HAL-UNIV-NANTES |
| Subject Terms: |
BERT; Transformers; Large Language Models; Clinical; Biomedical; Medical domain; Benchmarking; NLP evaluation; [INFO]Computer Science [cs]; [INFO.INFO-AI]Computer Science [cs]/Artificial Intelligence [cs.AI]; [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL]; [INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG]; [INFO.INFO-TT]Computer Science [cs]/Document and Text Processing |
| Subject Geographic: |
Torino; Italy |
| Description: |
International audience ; The recent emergence of Large Language Models (LLMs) has enabled significant advances in the field of Natural Language Processing (NLP). While these new models have demonstrated superior performance on various tasks, their application and potential are still underexplored, both in terms of the diversity of tasks they can handle and their domain of application. In this context, we evaluate four state-of-the-art instruction-tuned LLMs (ChatGPT, Flan-T5 UL2, Tk-Instruct, and Alpaca) on a set of 13 real-world clinical and biomedical NLP tasks in English, including named-entity recognition (NER), question-answering (QA), relation extraction (RE), and more. Our overall results show that these evaluated LLMs approach the performance of state-of-the-art models in zero- and few-shot scenarios for most tasks, particularly excelling in the QA task, even though they have never encountered examples from these tasks before. However, we also observe that the classification and RE tasks fall short of the performance achievable with specifically trained models designed for the medical field, such as PubMedBERT. Finally, we note that no single LLM outperforms all others across all studied tasks, with some models proving more suitable for certain tasks than others. |
| Document Type: |
conference object |
| Language: |
English |
| Availability: |
https://hal.science/hal-04470883; https://hal.science/hal-04470883v1/document; https://hal.science/hal-04470883v1/file/A%20Zero-shot%20and%20Few-shot%20Study%20of%20Instruction-Finetuned%20Large%20Language%20Models%20Applied%20to%20Clinical%20and%20Biomedical%20Tasks%20%281%29.pdf |
| Rights: |
https://creativecommons.org/publicdomain/zero/1.0/ ; info:eu-repo/semantics/OpenAccess |
| Accession Number: |
edsbas.9A335EA0 |
| Database: |
BASE |