Research

Multimodal Theory of Mind Reasoning

Reasoning about people’s mental states based on their behaviors observed from multimodal inputs.
• Multimodal ToM QA (ACL 2024) (Outstanding Paper Award)
• Multimodal Multiagent ToM (arXiv preprint)

Multimodal Embodied Assistance

Assisting humans with physical actions and verbal communication in complex embodied settings.
• Goal-Oriented Mental Alignment (IROS 2024)
• Neurally-guided Online Probabilistic Assistance (ICRA 2023)
• Watch-And-Help (ICLR 2021)

Learning from Human Feedback

Learning human perferences from rich human feedback.
• Pragmatic Feature Preferences (ICML 2024)
• Graph-based Active Reward Learning (ICML 2022)

NeurIPS 2023 Tutorial on Language Models Meet World Models

This tutorial aims to summarize and present a unified view of connecting WMs and LMs and highlight the various opportunities for improved machine reasoning and planning based on (or even beyond) large LMs through world modeling.

VirtualHome 2 (VirtualHome-Social)

An open-sourced mulit-agent household embodied simulator.