A Novel Data Augmentation Framework for Contrastive Self-Supervised Learning

Abstract

Self-supervised learning (SSL) has emerged as a leading paradigm in visual representation learning, employing techniques that encourage similarity among latent representations produced from diverse transformations of the same image. Despite the proven impact of data augmentation on such methods, the design of a suitable data augmentation pipeline for training such methods disregards the downstream tasks for which the learned representation is intended. To overcome this challenge, we introduce a novel data augmentation framework called Rank-Update (RU). This framework periodically assesses the relevance of different augmentation techniques for various downstream tasks, enabling the construction of an augmentation pipeline tailored to the specific requirements of those tasks without sacrificing generalization. This approach is appropriate in cases where we know the downstream tasks in advance but do not have enough data to train a model on these tasks directly through supervised learning. We first present an empirical study on the influence of different data augmentation techniques on SSL models across various downstream tasks. Furthermore, we then demonstrate the effectiveness of our approach in comparison to other methods on the same set of downstream tasks. Finally, we showcase the robustness of our method across different configurations, while presenting its limitations.