Using weight matrices to build a polynomial representation to assist with interpretability and explainability of deep learning models

Abstract

Deep Learning (DL) models use representational learning to discover the representation for classification and detection. Learned representations present a problem because the representation is encoded in such a way that cannot be explained using human intuition and known mathematics, thus not allowing us to reason out and interpret results from DL models. Knowledge learned by a neural network is internally stored in the weights. In this research project, we present a novel approach to building a polynomial representation of the learned function using the weight matrices. We were able to represent the learned function and also use convolution to map and to visualise the behaviour of the model. A popular explanation method, Local interpretable model agnostic explanations(LIME) was also used to obtain an understanding of predictions made using the Feedforward neural network (FFNN)