Mitigating class co-occurrence ambiguity in weakly supervised semantic segmentation with cross image attention

Abstract

Weakly Supervised Semantic Segmentation (WSSS) has gained traction as a cost effective alternative to fully supervised segmentation, which requires extensive pixel-level labels. The primary challenge in WSSS is generating accurate segmentation masks from only image-level labels. While various approaches have made progress, the issue of class co-occurrence, where certain classes frequently appear together within the same image, remains a significant obstacle. Unlike traditional methods that focus on separating objects from the background, we address the more challenging problem of disambiguating co-occurring classes, such as “baseball bat” and “baseball glove” or “mouse” and “keyboard”. We empirically demonstrate that class co-occurrence is a critical factor degrading segmentation performance, especially for datasets where co-occurrence is prevalent, such as MS COCO 2014. In contrast, the PASCAL VOC 2012 dataset, which exhibits less co-occurrence, shows minimal performance degradation. This discrepancy highlights that co-occurrence is a dataset-dependent challenge, overlooked in previous studies focusing primarily on PASCAL VOC 2012. To address this problem, we propose a novel dual-branch transformer architecture built upon MCTFormer+, where one branch processes the primary image and the other processes an auxiliary image containing a complementary class. The two branches interact via a new cross-attention mechanism called Cross Image Attention from the Auxiliary to Primary branch (CIA-AP), which selectively augments the primary image’s attention maps using information from the auxiliary image. This approach effectively improves segmentation accuracy for co-occurring classes by enhancing the model’s ability to distinguish between them. Furthermore, we introduce CCT Prime, a new contrastive learning module designed to align class token embeddings between the two branches, ensuring consistent representations for effective information transfer. This alignment proves essential for enhancing performance when co-occurring classes are present, while still maintaining competitive overall performance. Experiments show substantial improvements for co-occurring classes while maintaining competitive mean Intersection over Union (mIoU). Our findings highlight the effectiveness of leveraging cross-image information to address class co-occurrence, providing a promising direction for future WSSS research.