Social Cognitive AI (SCAI) Lab

Publications

embodied-ai theory-of-mind human-ai-cooperation multi-agent-cooperation social-learning social-cognition social-scene-understanding

Preprint

SIFToM: Robust Spoken Instruction Following through Theory of Mind
SIFToM: Robust Spoken Instruction Following through Theory of Mind
Lance Ying, Jason Xinyu Liu*, Shivam Aarya*, Yizirui Fang, Stefanie Tellex, Joshua B. Tenenbaum, Tianmin Shu
arXiv preprint
Paper
Website
Code
embodied-ai theory-of-mind human-ai-cooperation
On Benchmarking Human-Like Intelligence in Machines
On Benchmarking Human-Like Intelligence in Machines
Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum
arXiv preprint
Paper
theory-of-mind
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
Suyu Ye*, Haojun Shi*, Darren Shih, Hyokun Yun, Tanya Roosta, Tianmin Shu
arXiv preprint
Paper
Website
Code
human-ai-cooperation

2025

SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
Yan Zhuang, Jiawei Ren, Xiaokang Ye, Jianzhi Shen, Ruixuan Zhang, …, Xiyan Zhang, Ziqiao Ma, Lianhui Qin, Zhiting Hu, Tianmin Shu
NeurIPS 2025
Paper
Website
Code
embodied-ai
SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds
SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds
Xiaokang Ye*, Jiawei Ren*, Yan Zhuang, Xuhong He, Yiming Liang, …, Dhruv Vivek Sharma, Ziqiao Ma, Tianmin Shu, Zhiting Hu, Lianhui Qin
NeurIPS 2025 (Spotlight)
Paper
Website
embodied-ai
AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling
AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling
Zhining Zhang*, Chuanyang Jin*, Mung Yao Jia*, Shunchi Zhang*, Tianmin Shu
NeurIPS 2025 (Spotlight)
Paper
Website
Code
theory-of-mind
PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation
PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation
Yifan Yin*, Zhengtao Han*, Shivam Aarya, Jianxin Wang, Shuhang Xu, Jiawei Peng, Angtian Wang, Alan Yuille, Tianmin Shu
RSS 2025
Paper
Website
Code
embodied-ai
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
Hongxin Zhang*, Zeyuan Wang*, Qiushi Lyu*, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Behzad Dariush, Kwonjoon Lee, Yilun Du, Chuang Gan
ICLR 2025
Paper
Website
Code
embodied-ai multi-agent-cooperation
GenEx: Generating an Explorable World
GenEx: Generating an Explorable World
Taiming Lu, Tianmin Shu, Junfei Xiao, Luoxin Ye, Jiahao Wang, …, Chen Wei, Daniel Khashabi, Rama Chellappa, Alan Yuille, Jieneng Chen
ICLR 2025
Paper
Website
Code
embodied-ai
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
Haojun Shi*, Suyu Ye*, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, Tianmin Shu
AAAI 2025 (Oral)
Paper
Website
Code
theory-of-mind

2024

Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Weihua Du*, Qiushi Lyu*, Jiaming Shan, Zhenting Qi, Hongxin Zhang, …, Andi Peng, Tianmin Shu, Kwonjoon Lee, Behzad Dariush, Chuang Gan
NeurIPS 2024
Paper
Website
Code
embodied-ai human-ai-cooperation
Few-Shot Task Learning through Inverse Generative Modeling
Few-Shot Task Learning through Inverse Generative Modeling
Aviv Netanyahu, Yilun Du, Antonia Bronars, Jyothish Pari, Joshua Tenenbaum, Tianmin Shu, Pulkit Agrawal
NeurIPS 2024
Paper
Website
Code
embodied-ai social-learning
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
Lance Ying, Kunal Jha, Shivam Aarya, Joshua B. Tenenbaum, Antonio Torralba, Tianmin Shu
IROS 2024
Paper
Website
Code
embodied-ai theory-of-mind human-ai-cooperation
MMToM-QA: Multimodal Theory of Mind Question Answering
MMToM-QA: Multimodal Theory of Mind Question Answering
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer D. Ulman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu
ACL 2024 (Outstanding Paper Award)
Paper
Website
Code
Data
theory-of-mind
Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Andi Peng, Yuying Sun, Tianmin Shu, David Abel
ICML 2024
Paper
Code
social-learning
An asynchronous, automated workflow for looking time experiments with infants
An asynchronous, automated workflow for looking time experiments with infants
Gal Raz, Sabrina Piccolo, Janine Medrano, Shari Liu, Kirsten Lydic, Catherine Mei, Victoria Nguyen, Tianmin Shu, Rebecca Saxe
Developmental Psychology
Paper
theory-of-mind social-cognition
Building Cooperative Embodied Agents Modularly with Large Language Models
Building Cooperative Embodied Agents Modularly with Large Language Models
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan
ICLR 2024
Paper
Website
Code
embodied-ai human-agent-cooperation
Risk-Bounded Online Team Interventions via Theory of Mind
Risk-Bounded Online Team Interventions via Theory of Mind
Yuening Zhang, Paul Robertson, Tianmin Shu, Sungkweon Hong, Brian C. Williams
ICRA 2024
Paper
theory-of-mind human-ai-cooperation
Neural Amortized Inference for Nested Multi-agent Reasoning
Neural Amortized Inference for Nested Multi-agent Reasoning
Kunal Jha, Tuan Anh Le, Chuanyang Jin, Yen-Ling Kuo, Joshua B. Tenenbaum, Tianmin Shu
AAAI 2024
Paper
Website
Code
Supp
theory-of-mind

2023

Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning
Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning
Zhiting Hu*, Tianmin Shu*
arXiv preprent
Paper
Website
embodied-ai
Language Models Meet World Models: Embodied Experiences Enhance Language Models
Language Models Meet World Models: Embodied Experiences Enhance Language Models
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, Zhiting Hu
NeurIPS 2023
Paper
Code
embodied-ai
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Andi Peng, Aviv Netanyahu, Mark Ho, Tianmin Shu, Andreea Bobu, Julie Shah, Pulkit Agrawal
ICML 2023
Paper
Website
embodied-ai social-learning
Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Dianbo Liu*, Vedant Shah*, Oussama Boussif*, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Mozer, Nicolas Heess, Yoshua Bengio
ICLR 2023
Paper
Code
multi-agent-cooperation
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Xavier Puig*, Tianmin Shu*, Joshua B. Tenenbaum, Antonio Torralba
ICRA 2023
Paper
Website
Code
embodied-ai theory-of-mind human-ai-cooperation
Zero-shot linear combinations of grounded social interactions with Linear Social MDPs
Zero-shot linear combinations of grounded social interactions with Linear Social MDPs
Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum, Boris Katz, Andrei Barbu
AAAI 2023
Paper
theory-of-mind multi-agent-cooperation

2022

Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
Aviv Netanyahu*, Tianmin Shu*, Joshua B. Tenenbaum, Pulkit Agrawal
ICML 2022
Paper
Website
Code
social-learning
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, Zhiting Hu
EMNLP 2022
Paper
Code
Blog
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Xiaofeng Gao, Luyao Yuan, Tianmin Shu, Hongjing Lu, Song-Chun Zhu
IEEE Robotics and Automation Letters (RA-L)
Paper
Website
embodied-ai theory-of-mind human-ai-cooperation
Incorporating Rich Social Interactions Into MDPs
Incorporating Rich Social Interactions Into MDPs
Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum, Boris Katz, Andrei Barbu
ICRA 2022 (Excellent Paper Award at IROS Cognitive and Social Aspects of Human Multi-Robot Interaction Workshop, 2022)
Paper
Website
theory-of-mind social-scene-understanding

2021

Social Interactions as Recursive MDPs
Social Interactions as Recursive MDPs
Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Boris Katz, Andrei Barbu
CoRL 2021
Paper
Website
theory-of-mind social-scene-understanding
AGENT: A Benchmark for Core Psychological Reasoning
AGENT: A Benchmark for Core Psychological Reasoning
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin A. Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua B. Tenenbaum, Tomer D. Ullman
ICML 2021
Paper
Supp
Website
Code
Blog
theory-of-mind social-cognition
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, Antonio Torralba, Andrei Barbu
ICLR 2021 (Best Paper Award at NeurIPS Cooperative AI Workshop, 2020)
Paper
Website
Code
human-ai-cooperation embodied-ai theory-of-mind
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
Aviv Netanyahu*, Tianmin Shu*, Boris Katz, Andrei Barbu, Joshua B. Tenenbaum
AAAI 2021 (Best Paper Award at NeurIPS Shared Visual Representations in Human and Machine Intelligence Workshop, 2020)
Paper
Supp
Website
Code
social-scene-understanding theory-of-mind social-cognition
A Unified Psychological Space for Human Perception of Physical and Social Events
A Unified Psychological Space for Human Perception of Physical and Social Events
Tianmin Shu, Yujia Peng, Song-Chun Zhu, Hongjing Lu
Cognitive Psychology
Paper
Code
Data
social-scene-understanding social-cognition
Exploring Biological Motion Perception in Two-stream Convolutional Neural Networks
Exploring Biological Motion Perception in Two-stream Convolutional Neural Networks
Yujia Peng, Hannah Lee, Tianmin Shu, Hongjing Lu
Vision Research
Paper

2020

Adventures in Flatland: Perceiving Social Interactions Under Physical Dynamics
Adventures in Flatland: Perceiving Social Interactions Under Physical Dynamics
Tianmin Shu, Marta Kryven, Tomer D. Ullman, Joshua B. Tenenbaum
CogSci 2020
Paper
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Xiaofeng Gao*, Ran Gong*, Yizhou Zhao, Shu Wang, Tianmin Shu, Song-Chun Zhu
RO-MAN 2020
Paper
Website
Code
human-ai-cooperation theory-of-mind embodied-ai
Active Visual Information Gathering for Vision-Language Navigations
Active Visual Information Gathering for Vision-Language Navigations
Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, Jianbing Shen
ECCV 2020
Paper
Code
embodied-ai
Learning to Infer Human Attention in Daily Activities
Learning to Infer Human Attention in Daily Activities
Zhixiong Nan, Tianmin Shu, Ran Gong, Shu Wang, Ping Wei, Song-Chun Zhu, Nanning Zheng
Pattern Recognition
Paper
theory-of-mind

For papers published before 2020, please check out here.