Generalized Task Learning for Robots: Unifying Task Hierarchies through Contrastive Learning

Abstract

This dissertation addresses the challenge of enabling robots to generalize across unseen household tasks by learning abstract task structures from demonstration data. We develop a three-stage pipeline that translates natural language instructions and demonstrations into hierarchical task representations using large language models, clustering, and parameterized generalization. Our approach is tested and evaluated on the ALFRED benchmark [Shridhar et al. 2020]. ALFRED acts as a standardized measure used for training models to comprehend and follow instructions in natural language. It leverages first-person perspective visual input to carry out a series of actions for various household tasks. While this approach doesn’t represent the state-of-the-art, it establishes a foundation for future research to build upon.