| Title: |
Re-examining Concept-based Explainable Models for Multimodal Interpretative Tasks |
| Authors: |
Tores, Julie; Ancarani, Elisa; Sun, Rémy; Sassatelli, Lucile; Wu, Hui-Yin; Precioso, Frederic |
| Contributors: |
Modèles et algorithmes pour l’intelligence artificielle (MAASAI); Centre Inria d'Université Côte d'Azur; Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Laboratoire Jean Alexandre Dieudonné (LJAD); Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Scalable and Pervasive softwARe and Knowledge Systems (Laboratoire I3S - SPARKS); Laboratoire d'Informatique, Signaux, et Systèmes de Sophia Antipolis (I3S); Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Laboratoire d'Informatique, Signaux, et Systèmes de Sophia Antipolis (I3S); Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Centre National de la Recherche Scientifique (CNRS); Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA); Université Côte d'Azur (UniCA); Institut universitaire de France (IUF); Ministère de l'Education nationale, de l’Enseignement supérieur et de la Recherche (M.E.N.E.S.R.); Signal, Images et Systèmes (Laboratoire I3S - SIS); Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA)-Université Nice Sophia Antipolis (1965 - 2019) (UNS)-Centre National de la Recherche Scientifique (CNRS)-Université Côte d'Azur (UniCA); Biologically plausible Integrative mOdels of the Visual system : towards synergIstic Solutions for visually-Impaired people and artificial visiON (BIOVISION); Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria); ANR-21-CE38-0012,TRACTIVE,Vers une analyse multimodale automatique de l'esthétique discursive filmique(2021); European Project: 951911,H2020-ICT-2018-20,H2020-ICT-2019-3,AI4Media(2020) |
| Source: |
ACM digital library ; MM 2025 - 33rd ACM International Conference on Multimedia ; https://hal.science/hal-05462454 ; MM 2025 - 33rd ACM International Conference on Multimedia, Oct 2025, Dublin, Ireland. pp.12437-12445, ⟨10.1145/3746027.3758170⟩ ; https://dl.acm.org/doi/proceedings/10.1145/3746027 |
| Publisher Information: |
CCSD; ACM |
| Publication Year: |
2025 |
| Collection: |
HAL Université Côte d'Azur |
| Subject Terms: |
CCS Concepts; CCS Concepts Computing methodologies → Neural networks Scene understanding • Social and professional topics → User characteristics Video interpretation Multimodality Explainable AI Concept annotation; Concept annotation; Computing methodologies → Neural networks; Scene understanding; • Social and professional topics → User characteristics Video interpretation; Multimodality; Explainable AI; [INFO]Computer Science [cs] |
| Subject Geographic: |
Dublin; Ireland |
| Description: |
International audience ; Concept-based models have been proposed as a new line of research for explainable by-design deep learning models. However, those models show their whole power when applied to benchmarks where the concepts are well defined and the concepts' attributes easily extractable from the raw data. In this paper, we challenge the most recent concept-based model initially developed for image classification, on more complex interpretative tasks from a recently proposed video benchmark where they perform poorly. We conduct a root cause analysis of the poor performances of state-of-the-art explainable concept-based models for these multimodal interpretative tasks, and propose adaptations to design robust explainable models for detecting character objectification in this novel challenging video benchmark. We show that the optimal architectural choice may vary depending on the modality setting, thereby showing that designing multimodal concept-based approaches remains an open challenge and calls for further investigation. |
| Document Type: |
conference object |
| Language: |
English |
| Relation: |
info:eu-repo/grantAgreement//951911/EU/A European Excellence Centre for Media, Society and Democracy/AI4Media |
| DOI: |
10.1145/3746027.3758170 |
| Availability: |
https://hal.science/hal-05462454; https://hal.science/hal-05462454v1/document; https://hal.science/hal-05462454v1/file/3746027.3758170.pdf; https://doi.org/10.1145/3746027.3758170 |
| Rights: |
https://creativecommons.org/licenses/by/4.0/ ; info:eu-repo/semantics/OpenAccess |
| Accession Number: |
edsbas.C7A0CF5A |
| Database: |
BASE |