| Title: |
Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models |
| Authors: |
Pacaud, Paul; Garcia, Ricardo; Chen, Shizhe; Schmid, Cordelia |
| Contributors: |
Models of visual object recognition and scene understanding (WILLOW); Département d'informatique - ENS-PSL (DI-ENS); École normale supérieure - Paris (ENS-PSL); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-École normale supérieure - Paris (ENS-PSL); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Centre Inria de Paris; Institut National de Recherche en Informatique et en Automatique (Inria); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS); This work was performed using HPC resources from GENCI-IDRIS (Grant 2025-AD011015795 and AD011015795R1). It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “France 2030” program, reference ANR-23-IACL-0008 (PR AI RIE-PSAI project), the ANR project VideoPredict ANR-21-FAI1-0002- 01. Cordelia Schmid would like to acknowledge the support by the Korber European Science Prize.; ANR-23-IACL-0008,PR AI RIE-PSAI,PR AI RIE-PSAI - Paris School of Artificial Intelligence(2023); ANR-21-FAI1-0002,VideoPredict,Predire l'avenir video(2021) |
| Source: |
https://hal.science/hal-05392773 ; 2025. |
| Publisher Information: |
CCSD |
| Publication Year: |
2025 |
| Subject Terms: |
Vision-Language Model; Robotic Manipulation; Failure Detection and Recovery; Robotics; Computer Vision and Pattern Recognition; Artificial intelligence; ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE; ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE/I.2.9: Robotics; ACM: I.: Computing Methodologies/I.4: IMAGE PROCESSING AND COMPUTER VISION; [INFO.INFO-AI]Computer Science [cs]/Artificial Intelligence [cs.AI]; [INFO.INFO-CV]Computer Science [cs]/Computer Vision and Pattern Recognition [cs.CV]; [INFO.INFO-RB]Computer Science [cs]/Robotics [cs.RO] |
| Description: |
Robust robotic manipulation requires reliable failure detection and recovery. Although current Vision-Language Models (VLMs) show promise, their accuracy and generalization are limited by the scarcity of failure data. To address this data gap, we propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures. This method produces not only binary classification labels but also fine-grained failure categories and step-by-step reasoning traces in both simulation and the real world. With it, we construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail, substantially expanding the diversity and scale of existing failure datasets. We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection. Guardian achieves state-of-the-art performance on both existing and newly introduced benchmarks. It also effectively improves task success rates when integrated into a state-of-the-art manipulation system in simulation and real robots, demonstrating the impact of our generated failure data. |
| Document Type: |
report |
| Language: |
English |
| Relation: |
info:eu-repo/semantics/altIdentifier/arxiv/2512.01946; ARXIV: 2512.01946 |
| Availability: |
https://hal.science/hal-05392773; https://hal.science/hal-05392773v1/document; https://hal.science/hal-05392773v1/file/2025-11_Guardian.pdf |
| Rights: |
https://hal.science/licences/copyright/ ; info:eu-repo/semantics/OpenAccess |
| Accession Number: |
edsbas.8238D66 |
| Database: |
BASE |