Evaluating elastic weight consolidation across a taxonomy of continual learning scenarios
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of the Witwatersrand, Johannesburg
Abstract
The dynamic nature of real-world datasets poses significant challenges for machine learning, especially in continual learning, where catastrophic forgetting occurs as neural networks forget prior knowledge when acquiring new information, known as catastrophic forgetting. Various methods have been investigated in order to overcome catastrophic forgetting, such as Elastic Weight Consolidation (EWC). This study aims to determine the effectiveness of EWC in diverse real-world datasets characterised by different continual learning scenarios. We compare EWC against a base model with no regularisation, models with different continual learning approaches, and a method that incorporates data rehearsal from the previous task. Through experiments and performance comparisons using datasets specifically designed to represent three key iterated learning scenarios—class-iterated learning, domain-iterated learning, and task-iterated learning—this research aims to provide a comprehensive assessment of the capacity of EWC to facilitate continual learning. To ensure a thorough evaluation, these scenarios were carefully selected to include a toy domain, an MNIST-based domain, and a practical real-world domain, reflecting a progression from abstract conceptual tasks to more complex and realistic challenges. This meticulous approach highlights the core contribution of the research, emphasising its relevance and applicability across varying levels of complexity in continual learning. Our findings show that while EWC consistently delivers stable performance across diverse continual learning scenarios, it struggles with adaptability in tasks involving high feature overlap or dynamic shifts. In contrast, Synaptic Intelligence offers greater flexibility but exhibits higher variability, especially when task boundaries are ambiguous or overlap occurs at deeper representation levels.
Description
A research report submitted in fulfilment of the requirements for the degree of Master of Science in Data Science, to the Faculty of Science, School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, 2025
Citation
Laxton, Gregory Stephen. (2025). Evaluating elastic weight consolidation across a taxonomy of continual learning scenarios. [Master's dissertation, University of the Witwatersrand, Johannesburg]. WIReDSpace. https://hdl.handle.net/10539/50305