Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation

We propose CaPE (Code as Path Editor), a safe and interpretable multimodal path planning framework for multi-agent cooperation. CaPE enables robots to adapt motion plans in response to natural language communication by generating structured, interpretable path-editing programs that are validated by a model-based planner.

• 🧭 Planner + editor: A model-based joint planner proposes candidate paths for the target agent and predicts other agents’ trajectories. CaPE then edits the target path using language and contextual information.
• 🧩 Language-guided path editing: Cooperation is formulated as editing an existing plan. CaPE selects among planner-proposed candidates and applies targeted waypoint-level edits.
• 👁️🗣️ Multimodal program synthesis: A vision-language model generates a compact DSL program conditioned on the map, candidate paths, predicted agent trajectories, and communication.
• ✅ Safety verification: A planner-based verifier checks each DSL instruction and rejects edits that violate physical or interaction constraints, producing a verified final path.
• 🔎 Interpretability: The output is explicit, human-readable edit code, such as path selection, waypoint translation or rotation, waypoint insertion, and waiting.

Core idea: Use a vision-language model to synthesize structured path-editing programs, then apply planner-based verification to ensure safety. This enables open-ended language-driven coordination while preserving robustness and interpretability.