CSE Colloquia: Lipstick on a Pig: Using Language Models as Few-Shot Learners

Abstract: Language models provide representations that can be adapted to many NLP tasks with minimal effort. In particular, language modeling has provided exceptional few-shot natural language understanding and reasoning performance by framing these tasks as prompts. These capabilities improve significantly with larger models and datasets, making the direct use of language models the dominant approach for NLP applications. However, the goals of language modeling are not precisely the same as what we need from few-shot learners, and it is vital to understand this gap.

In this talk, we will discuss the basics of neural networks, the text corpus, and the training pipeline that enables language models to behave as these general-purpose AI agents. However, we will show how this very paradigm of language modeling also introduces fundamental limitations in this technology. We will characterize these vulnerabilities in language models and discuss how they affect end-use applications. The results presented suggest language modeling may not be sufficient to learn robust reasoners and that we need to take the pretraining data into account when interpreting few-shot evaluation results.

Bio: Dr. Sameer Singh is an Associate Professor of Computer Science at the University of California, Irvine (UCI) and a Cofounder of Spiffy AI. He is working primarily on the robustness and interpretability of machine learning algorithms and models that reason with text and structure for natural language processing. He has been named the Kavli Fellow by the National Academy of Sciences, received the NSF CAREER award, UCI Distinguished Early Career Faculty award, the Hellman Faculty Fellowship, and was selected as a DARPA Riser. His group has received funding from Allen Institute for AI, Amazon, NSF, DARPA, Adobe Research, Hasso Plattner Institute, NEC, Base 11, and FICO. Sameer has published extensively at machine learning and natural language processing venues and received numerous paper awards, including at KDD 2016, ACL 2018, EMNLP 2019, AKBC 2020, ACL 2020, and NAACL 2022. (https://sameersingh.org/)