| Title: |
Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation |
| Authors: |
Chen, Shizhe; Garcia, Ricardo; Pacaud, Paul; Schmid, Cordelia |
| Contributors: |
Models of visual object recognition and scene understanding (WILLOW); Département d'informatique - ENS-PSL (DI-ENS); École normale supérieure - Paris (ENS-PSL); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-École normale supérieure - Paris (ENS-PSL); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Centre Inria de Paris; Institut National de Recherche en Informatique et en Automatique (Inria); Université Paris Sciences et Lettres (PSL)-Université Paris Sciences et Lettres (PSL)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS); This work was partially supported by the HPC resources from GENCI-IDRIS (Grant 20XXAD011012122 and AD011014846). It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “France 2030” program, reference ANR-23-IACL-0008 (PR AI RIE-PSAI projet), the ANR project VideoPredict (ANR-21-FAI1-0002-01), and the Paris Ile-de-France Region in the frame of the DIM AI4IDF.; ANR-23-IACL-0008,PR AI RIE-PSAI,PR AI RIE-PSAI - Paris School of Artificial Intelligence(2023); ANR-21-FAI1-0002,VideoPredict,Predire l'avenir video(2021) |
| Source: |
https://hal.science/hal-05391955 ; 2025. |
| Publisher Information: |
CCSD |
| Publication Year: |
2025 |
| Subject Terms: |
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO); Vision-Language Model; Task Planning; Robotic Manipulation; ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE; ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE/I.2.9: Robotics; ACM: I.: Computing Methodologies/I.4: IMAGE PROCESSING AND COMPUTER VISION; [INFO]Computer Science [cs]; [INFO.INFO-CV]Computer Science [cs]/Computer Vision and Pattern Recognition [cs.CV]; [INFO.INFO-RB]Computer Science [cs]/Robotics [cs.RO] |
| Description: |
Robotic manipulation faces a significant challenge in generalizing across unseen objects, environments and tasks specified by diverse language instructions. To improve generalization capabilities, recent research has incorporated large language models (LLMs) for planning and action execution. While promising, these methods often fall short in generating grounded plans in visual environments. Although efforts have been made to perform visual instructional tuning on LLMs for robotic manipulation, existing methods are typically constrained by single-view image input and struggle with precise object grounding. In this work, we introduce Gondola, a novel grounded vision-language planning model based on LLMs for generalizable robotic manipulation. Gondola takes multi-view images and history plans to produce the next action plan with interleaved texts and segmentation masks of target objects and locations. To support the training of Gondola, we construct three types of datasets using the RLBench simulator, namely robot grounded planning, multi-view referring expression and pseudo long-horizon task datasets. Gondola outperforms the state-of-the-art LLM-based method across all four generalization levels of the GemBench dataset, including novel placements, rigid objects, articulated objects and long-horizon tasks. |
| Document Type: |
report |
| Language: |
English |
| Relation: |
info:eu-repo/semantics/altIdentifier/arxiv/2506.11261; ARXIV: 2506.11261 |
| DOI: |
10.48550/arXiv.2506.11261 |
| Availability: |
https://hal.science/hal-05391955; https://hal.science/hal-05391955v1/document; https://hal.science/hal-05391955v1/file/2025-06_Gondola.pdf; https://doi.org/10.48550/arXiv.2506.11261 |
| Rights: |
https://creativecommons.org/licenses/by/4.0/ ; info:eu-repo/semantics/OpenAccess |
| Accession Number: |
edsbas.9F03F590 |
| Database: |
BASE |