CSE Colloquium: Learning to Connect Images and Text for Natural Communication

Abstract: From the gestures that accompany speech to images in social media posts, humans effortlessly combine words with visual presentations. However, machines are not equipped to understand and generate such presentations due to people’s pervasive reliance on common-sense and world knowledge in relating words and images. I present a novel framework for modeling and learning a deeper combined understanding of text and images by classifying inferential relations to predict temporal, causal and logical entailments in context. This enables systems to make inferences with high accuracy while revealing author expectations and social-context preferences. I go on to design machine learning models for generating text based on visual input that use these inferences to provide users with key requested information. The results show a dramatic improvement in the consistency and quality of the generated text by decreasing spurious information by half. Finally, I briefly sketch my other projects on multimodal and communicative systems and describe my research vision: to build human-level collaborative artificial intelligence entities by leveraging the cognitive science of language use. 

Biography: Malihe Alikhani is a 5th year Ph.D. candidate in the department of computer science at Rutgers University, advised by Prof. Matthew Stone. She is pursuing a certificate in cognitive science through the Rutgers Center for Cognitive Science and holds a BA and MA in Mathematics. Her research aims at teaching machines to understand and generate multimodal communication. She is the recipient of the fellowship award for excellence in computation and data sciences from the Rutgers Discovery Informatics Institute in 2018 and the Anita Borg student fellowship in 2019. Before joining Rutgers, she was a lecturer and an adjunct professor of Mathematics and Statistics for a year at San Diego State University and San Diego Mesa College. She has organized workshops and tutorials at ACL and EMNLP and has served as the program committee of several conferences and journals including ACL and AAAI conferences and is currently the associate editor of the Mental Note Journal.