CSE Colloquium: Communicating with Embodied Agents

Abstract:
Despite recent advances, language communication in embodied AI still faces many challenges. Human language not only needs to ground to agents’ perception and action but also needs to facilitate collaboration between humans and agents.  To address these challenges, I will introduce several efforts in my lab that study pragmatic communication with embodied agents. I will talk about cognitively motivated grounded language learning that facilitates fast mapping. I will discuss task learning by following language instructions and highlight the importance of physical commonsense reasoning. I will further present explicit modeling of partners’ goals, beliefs, and abilities (i.e., theory of mind) and discuss its role in language communication for situated collaborative tasks.

Bio:
Joyce Chai is a Professor in the Department of Electrical Engineering and Computer Science at the University of Michigan. She holds a Ph.D. in Computer Science from Duke University. Her research interests span from natural language processing and embodied AI to human-AI collaboration. She is fascinated by how experience with the world and how social pragmatics shape language learning and language use; and is excited about developing language technology that is sensorimotor grounded, pragmatically rich, and cognitively motivated. Her current work explores the intersection between language, perception, and action to enable situated communication with embodied agents. She served on the executive board of NAACL and as Program Co-Chair for multiple conferences – most recently ACL 2020. She is a recipient of the NSF Career Award and several paper awards with her students (e.g., Best Long Paper Award at ACL 2010, Outstanding Paper Awards at EMNLP 2021 and ACL 2023). She is a Fellow of ACL.