Following Instructions and Asking Questions

Abstract

As we move towards the creation of embodied agents that understand natural language, several new challenges and complexities arise for grounding (e.g. complex state spaces), planning (e.g. long horizons), and social interaction (e.g. asking for help or clarifications). In this talk, I’ll discuss several recent results both on improvements to embodied instruction following within ALFRED and initial steps towards building agents that ask questions or model theory-of-mind.

Bio

Yonatan Bisk is an Assistant Professor at Carnegie Mellon University. His research area is Natural Language Processing (NLP) with a focus on grounding and embodiment. In particular, his interests are broadly: Modeling the semantics of the physical world, and Connecting language to perception and control.